Introduction

Any ‘Swiftie’ would be able to name the title of a Taylor Swift track just by listening to a few first seconds of it. I targeted whether a machine could do so if trained to be a Swiftie. However, to ease this task, I proposed to test whether I could have my machine predict the theme of a Taylor Swift song.

The goal of this project is to build a machine learning model that can predict the theme of a Taylor Swift song based on frequently used words in only the first 30% of lyrics. Using data from Spotify API and Musixmatch API, where Spotify obtains its lyrical data, and implementing multiple techniques to yield the most accurate model for this unsupervised machine learning project, let’s dive into our text analysis.

Background Information

As Spotify’s Global Artist of 2023, researching Taylor Swift data on Spotify seemed extremely pertinent, especially with her current, ongoing Eras Tour that has become the highest-grossing tour of all time, surpassing $1 billion in revenue. Renowned as one of the most successful artists of the 21st century, Swift is known for influencing the music industry by crossing genres: country, pop, alternative rock, indie folk, and electronic; challenging the industry norm (such as removing her Spotify catalog in 2013 to urge the platform in paying its artists more per stream); and her unmatchable artistry and lyricism.

In 2019, Swift came in dispute with her former record label, leading her to re-record her first six studio albums: Taylor Swift, Fearless, Speak Now, RED, 1989 and reputation. Since 2021, Swift began her re-recording journey with Fearless (Taylor’s Version), in which her re-recorded albums and songs are now retitled with (Taylor’s Version) along with cut songs that she did not release at the time denoted as (From The Vault) tracks. Her re-recordings have been critically acclaimed and broken countless sales, streaming, and chart records, especially with the latest re-record 1989 (Taylor’s Version).

Motive

I intend to study Swift’s owned discography in my project (with the exception of Taylor Swift and reputation which have not been released at the time of the project) within Spotify/Musixmatch’s catalog.

What Dataset?

I will be creating a dataset from data from Spotify API for Developers to analyze Swift’s discography and Musixmatch API to extract Swift’s song lyrics for each corresponding track. I will be performing machine learning on Musixmatch API to analyze Swift’s lyrics and musicality. Disclaimer: In accordance with Spotify Developer Policy, no Spotify Platform nor Spotify Content was used to train a machine learning or AI model nor ingested into a machine learning or AI model

Data Collection

Spotify API

The Spotify Web API provides a range of functionality for developers, such as the ability to retrieve data from artists, albums or shows; search for Spotify content; and so forth. Using the spotifyr package, I can search for and extract all tracks (including albums and singles) on Spotify with Swift listed as the artist through the get_artist_audio_features() function.

Further, Spotify’s API enables me access to sound features such as: danceability, energy, key, loudness, mode, speechiness, acousticness, instrumentalness, liveness, etc. for each track.

Song Selection

Using Spotify’s catalog as of January 2024, we browse through Swift’s singles and albums on Spotify. We will analyze all distinct songs that Swift own: Taylor’s Version if possible and not including remixes.

When filtering our the songs, we choose all songs post 2017 (i.e after the release of reputation), in which Swift began owning and reclaiming her catalogue. A condition on album release date to include her debut album, released in 2006, was also included since it has yet to be re-recorded. Further, we select only the studio recording of songs that may have been released in alternate formats or live editions, such as the Folklore Long Pond Studio Sessions or Lover Live From Paris. In addition, we filter out remixes, singles, acoustic/piano versions, etc. Some alternative versions of songs are included, however, such as the multiple variations of All Too Well, Snow on the Beach, or the lakes.

ts_albums <- get_artist_audio_features(artist = 'taylor swift',
                                include_groups = "album")

ts_singles <- get_artist_audio_features(artist = 'taylor swift',
                                include_groups = "single")

ts_discography <- full_join(ts_albums %>%
  select(track_name, track_number, album_name, album_type, album_release_date, 
         danceability, energy, key, loudness, mode, speechiness, 
         acousticness, instrumentalness, liveness, valence, tempo,
         time_signature, key_mode, duration_ms, explicit), 
  ts_singles %>%
  select(track_name, track_number, album_name, album_type, album_release_date, 
         danceability, energy, key, loudness, mode, speechiness, 
         acousticness, instrumentalness, liveness, valence, tempo,
         time_signature, key_mode, duration_ms, explicit)) %>%
  arrange(desc(album_release_date))

ts <- ts_discography %>%
  filter(album_release_date > '2017-11-09' | album_release_date == '2006-10-24' |
           track_name == 'Sweeter Than Fiction - From "One Chance" Soundtrack') %>%
  filter(album_name != 'Spotify Singles' & !grepl('witch', album_name) & !grepl('Remix', track_name) & !grepl('the long pond studio sessions', track_name) & !grepl('Live From Paris', track_name) & !grepl('Video Edition', track_name) & !grepl('Live from the 2020 Academy of Country Music Awards', track_name) & !grepl('Recorded Live at the 2019 iHeartRadio Jingle Ball', track_name) & !grepl('cabin in candlelight version', track_name) & !grepl('Love Story - Pop Mix', track_name) & !grepl('Sawyr And Ryan Tedder Mix', track_name) & !grepl("willow - 90's trend remix", track_name) & !grepl('Acoustic Version', track_name) & !grepl('Piano', track_name)) %>%
  distinct(track_name, .keep_all = T)

ts[ts$album_name == 'folklore: the long pond studio sessions (from the Disney+ special) [deluxe edition]', 'album_name'] <- 'folklore (deluxe version)'

A preview of the Spotify API dataset is shown below:

track_name	track_number	album_name	album_type	album_release_date	danceability	energy	key	loudness	mode	speechiness	acousticness	instrumentalness	liveness	valence	tempo	time_signature	key_mode	duration_ms	explicit
I’m Only Me When I’m With You	12	Taylor Swift	album	2006-10-24	0.563	0.934	8	-3.629	1	0.0646	0.004520	8.07e-04	0.1030	0.518	143.964	4	G# major	213053	FALSE
Blank Space (Taylor’s Version)	2	1989 (Taylor’s Version) [Deluxe]	album	2023-10-27	0.733	0.733	0	-5.376	1	0.0670	0.088500	0.00e+00	0.1680	0.701	96.057	4	C major	231833	FALSE
I Did Something Bad	3	reputation	album	2017-11-10	0.696	0.602	0	-6.156	0	0.1590	0.067900	2.11e-05	0.0696	0.305	82.989	4	C minor	238253	FALSE
I Bet You Think About Me (feat. Chris Stapleton) (Taylor’s Version) (From The Vault)	26	Red (Taylor’s Version)	album	2021-11-12	0.391	0.715	0	-4.516	1	0.0495	0.167000	0.00e+00	0.1830	0.473	149.654	3	C major	285106	TRUE
The Joker And The Queen (feat. Taylor Swift)	1	The Joker And The Queen (feat. Taylor Swift)	single	2022-02-11	0.528	0.309	0	-7.398	1	0.0329	0.924000	2.10e-06	0.2820	0.307	133.830	4	C major	185422	FALSE
Sweeter Than Fiction - From “One Chance” Soundtrack	1	Sweeter Than Fiction	single	2013-10-21	0.710	0.614	6	-6.965	1	0.0366	0.001640	1.54e-04	0.0785	0.673	135.012	4	F# major	237640	FALSE
All You Had To Do Was Stay (Taylor’s Version)	5	1989 (Taylor’s Version) [Deluxe]	album	2023-10-27	0.588	0.721	0	-5.579	1	0.0317	0.000656	0.00e+00	0.1310	0.520	96.997	4	C major	193289	FALSE
I Think He Knows	6	Lover	album	2019-08-23	0.897	0.366	0	-8.029	1	0.0569	0.008890	3.53e-04	0.0715	0.416	100.003	4	C major	173386	FALSE
evermore (feat. Bon Iver)	15	evermore (deluxe version)	album	2021-01-07	0.390	0.270	1	-10.673	1	0.0308	0.937000	2.27e-03	0.1110	0.320	125.177	5	C# major	304106	FALSE
That’s When (feat. Keith Urban) (Taylor’s Version) (From The Vault)	24	Fearless (Taylor’s Version)	album	2021-04-09	0.588	0.608	5	-7.062	1	0.0365	0.225000	0.00e+00	0.0920	0.508	90.201	4	F major	189495	FALSE
High Infidelity	17	Midnights (The Til Dawn Edition)	album	2023-05-26	0.564	0.545	10	-10.170	1	0.1040	0.726000	0.00e+00	0.0878	0.766	176.082	4	A# major	231475	FALSE
I Knew You Were Trouble (Taylor’s Version)	4	Red (Taylor’s Version)	album	2021-11-12	0.584	0.557	6	-6.371	1	0.0342	0.012900	0.00e+00	0.0576	0.767	154.008	4	F# major	219760	FALSE
Today Was A Fairytale (Taylor’s Version)	20	Fearless (Taylor’s Version)	album	2021-04-09	0.482	0.779	7	-5.025	1	0.0409	0.013500	0.00e+00	0.2960	0.328	158.162	4	G major	241822	FALSE

Musixmatch API

The Musixmatch API allows developers to read objects from their licesnced, lyrics database. After creating an access token to request Musixmatch’s API, I retrieve starter code from fernandabruno/musixmatchR on Github to help me extract the data. Here I create two helper functions: searchTrack() and getLyrics() which will respectively retrieve the unique Musixmatch track IDs for an artist’s songs and then access the lyrics to that corresponding ID.

searchTrack <- function(artist, songName, api_key) {
  require(jsonlite)
  artist <- gsub(" ", "%20", artist)
  song <- gsub(" ", "%20", songName)
  data <- jsonlite::fromJSON(paste0(
    "http://api.musixmatch.com/ws/1.1/track.search?q_artist=", artist, "&q_track=", song, "&apikey=", api_key))
  
  trackInfo <- data.frame(track_id = unlist(data$message$body$track_list$track$track_id),
                       track_name = data$message$body$track_list$track$track_name,
                       album_name = data$message$body$track_list$track$album_name,
                       album_id = data$message$body$track_list$track$album_id,
                       artist_name = data$message$body$track_list$track$artist_name,
                       explicit = data$message$body$track_list$track$explicit,
                       stringsAsFactors = FALSE)
  return(trackInfo)
} #end

getLyrics <- function(trackID, api_key){
  require(jsonlite)
  metadata <- jsonlite::fromJSON(paste0(
    "http://api.musixmatch.com/ws/1.1/track.lyrics.get?track_id=", trackID, "&apikey=", api_key))

tracks <- data.frame(track_id = trackID,
                     lyrics_body = gsub("[\r\n]", " ",  metadata$message$body$lyrics$lyrics_body))
  return(tracks)
} # end

To extract the Musixmatch lyrics with my helper functions, I need to rely on the track_id information for each track, as designated by Musixmatch. I create a data frame test which will contain all of the track names and track ID’s in Musixmatch’s catalog.

In addition, Musixmatch’s catalog includes piano covers and instrumental versions of Taylor Swift songs; such inclusions requires filtering and using the distinct() function to select the specific songs only. In addition, select songs need direct handling due to their particular naming convention in Musixmatch- which we respond to by adding them to our dataset manually. After garnering Swift’s 219 songs, I use the getLyrics() function in order to extract the lyrics for each corresponding track id.

# We create a `test` data frame which holds all of the track names and track id's in Musixmatch's catalog
test <- data.frame()
for (i in ts$track_name) {
  test <- rbind(test, searchTrack('taylor swift', i, api_key))
}

# We check what tracks are stored with different naming conventions between the two datasets
test %>%
  filter(!(track_name %in% ts$track_name)) %>%
  distinct(track_name)

# we filter out the repeated track titles (and also songs in Musixmatch that are covers or instrumental versions which we don't want to consider)
test <- test %>%
  filter((track_name %in% ts$track_name)) %>%
  distinct(track_name, .keep_all = T) 

test <- test %>% 
  rbind(test, searchTrack('taylor swift', 'Sweeter Than Fiction', api_key)) %>% 
  rbind(test, searchTrack('taylor swift', "it's time to go (bonus track)", api_key)) %>% 
  rbind(test, searchTrack('taylor swift', "right where you left me (bonus track)", api_key)) %>% 
  rbind(test, searchTrack('taylor swift', "Only The Young (Featured in Miss Americana)", api_key)) %>% 
  rbind(test, searchTrack('taylor swift', "'tis the damn season", api_key)) %>% 
  rbind(test, searchTrack('taylor swift', "Lavender Haze (Felix Jaehn Remix)", api_key)) %>% 
  rbind(test, searchTrack('taylor swift', "Nothing New (Taylor's Version) (From The Vault) [feat. Phoebe Bridgers]", api_key)) %>% 
  rbind(test, searchTrack('taylor swift', "I Bet You Think About Me (Taylor's Version) (From The Vault) [feat. Chris Stapleton]", api_key)) %>% 
  distinct(track_name, .keep_all = T) %>% 
  filter(track_id != 269664185 & track_id != 253984170) %>% 
  filter(track_id != 258853439 & track_id != 226322338 & track_id != 258853440)

# we use our getLyrics() function to find the lyrics using the track_id in our `test` dataset 
lyrics <- data.frame()
for (i in test$track_id) {
  lyrics <- rbind(lyrics, getLyrics(i, api_key))
}

Data Creation

Having two different datasets, test and lyrics for the track names and lyrics, I can merge the Musixmatch data together into one dataset on track_id. Afterwards, I want to combine this newly merged lyrics dataset with the ts dataset from Spotify API.

However, there exists some differences in naming conventions of track titles between Spotify and Musixmatch, such as differences in how to label featured artists, whether using () or [] for example. In adherence with how the track names are displayed on Spotify, I explicitly rename the titles in the lyrics data frame, e.g. Electric Touch (Taylor’s Version) (From The Vault) [feat. Fall Out Boy] or Run (Taylor’s Version) (From The Vault) [feat. Ed Sheeran].

Having the track names be consistent between the datasets, I can merge on the track name and now have one singular dataset: official_list which will have the following inputs: track_name, album_name, track_number, lyrics, and length. I store this compiled dataset as a .csv file for convenience of analysis repetition.

# we merge test and lyrics
lyrics <- merge(lyrics, test, by = 'track_id')

# we check the naming conventions between each dataset
lyrics %>%
  filter(!(track_name %in% ts$track_name))
ts %>%
  filter(!track_name %in% lyrics$track_name)

# we fix the naming conventions (we adhere to the names that they are given in the Spotify catalog)
lyrics[lyrics$track_name == "Electric Touch (Taylor’s Version) (From The Vault) [feat. Fall Out Boy]", 'track_name'] <- "Electric Touch (feat. Fall Out Boy) (Taylor’s Version) (From The Vault)"
lyrics[lyrics$track_name == "Castles Crumbling (Taylor’s Version) (From The Vault) [feat. Hayley Williams]", 'track_name'] <- "Castles Crumbling (feat. Hayley Williams) (Taylor’s Version) (From The Vault)"
lyrics[lyrics$track_name == "Run (Taylor's Version) (From The Vault) [feat. Ed Sheeran]", 'track_name'] <- "Run (feat. Ed Sheeran) (Taylor’s Version) (From The Vault)"
lyrics[lyrics$track_name == "I Bet You Think About Me (Taylor's Version) [From The Vault] [feat. Chris Stapleton]", 'track_name'] <- "I Bet You Think About Me (feat. Chris Stapleton) (Taylor’s Version) (From The Vault)"
lyrics[lyrics$track_name == "Nothing New (Taylor's Version) (From The Vault) [feat. Phoebe Bridgers]", 'track_name'] <- "Nothing New (feat. Phoebe Bridgers) (Taylor’s Version) (From The Vault)"
lyrics[lyrics$track_name == "Everything Has Changed (Taylor's Version) [feat. Ed Sheeran]", 'track_name'] <- "Everything Has Changed (feat. Ed Sheeran) (Taylor’s Version)"
lyrics[lyrics$track_name == "The Last Time (Taylor's Version) [feat. Gary Lightbody]", 'track_name'] <- "The Last Time (feat. Gary Lightbody of Snow Patrol) (Taylor’s Version)"
lyrics[lyrics$track_name == "it's time to go (bonus track)", 'track_name'] <- "it’s time to go - bonus track"
lyrics[lyrics$track_name == "right where you left me (bonus track)", 'track_name'] <- "right where you left me - bonus track"
lyrics[lyrics$track_name == "'tis the damn season", 'track_name'] <- "‘tis the damn season"
lyrics[lyrics$track_name == "Only The Young (Featured in Miss Americana)", 'track_name'] <- "Only The Young - Featured in Miss Americana"
lyrics[lyrics$track_name == "Sweeter Than Fiction", 'track_name'] <- 'Sweeter Than Fiction - From "One Chance" Soundtrack'  
lyrics[lyrics$track_name == "Lavender Haze (Felix Jaehn Remix)", 'track_name'] <- 'Lavender Haze'
lyrics[lyrics$track_name == "   
Message In A Bottle (Taylor's Version) [From The Vault]", 'track_name'] <- "Message In A Bottle (Taylor's Version) (From The Vault)"
lyrics[lyrics$track_name == "Forever Winter (Taylor's Version) [From The Vault]", 'track_name'] <- "Forever Winter (Taylor's Version) (From The Vault)"

# we finally curate our official taylor swift catalog, with the track name, album name, track number, lyrics, and length
official_list <- merge(lyrics, ts, by = 'track_name') %>%
  select('track_name', 'album_name.y', 'track_number', 'lyrics_body', 'duration_ms')

About official_list (more information in codebook):

track_name: the title of the song
album_name: the title of the album, extended play, compilation, etc. that the track derives from
track_number: the number of the song in its album, extended play, compilation, etc. (note that if the track is a single, the track number is just 1)
lyrics: the first 30% of the song lyrics from Musixmatch’s API dataset (the dataset only provdes 30% of lyrics for non-subscription users)
length: the time duration of the song in milliseconds

A preview of our dataset:

track_name	album_name.y	track_number	lyrics_body	duration_ms
Karma (feat. Ice Spice)	Midnights (The Til Dawn Edition)	23	Karma is that girl, like (grah) You’re talkin’ shit for the hell of it Addicted to betrayal but you’re relevant You’re terrified to look down ’Cause if you dare, you’ll see the glare Of everyone you burned just to get there It’s coming back around And I keep my side of the street clean You wouldn’t know what I mean ’Cause karma is my boyfriend Karma is a god Karma is the breeze in my hair on the weekend Karma’s a relaxing thought Aren’t you envious that for you it’s not? Sweet like honey, karma is a cat Purring in my lap ’cause it loves me Flexing like a goddamn acrobat Me and karma vibe like that Karma is your checks ’bout to bounce (damn) Karma is the fire in your house (grah) And she ’bout to pop up unannounced (like) … ***** This Lyrics is NOT for Commercial use ***** (1409624132591)	201493
peace	folklore (deluxe version)	15	Our coming-of-age has come and gone Suddenly the summer, it’s clear I never had the courage of my convictions As long as danger is near And it’s just around the corner, darling ’Cause it lives in me No, I could never give you peace But I’m a fire, and I’ll keep your brittle heart warm If your cascade ocean wave blues come All these people think love’s for show But I would die for you in secret The devil’s in the details, but you got a friend in me … ***** This Lyrics is NOT for Commercial use ***** (1409624132591)	234000
Don’t You (Taylor’s Version) (From The Vault)	Fearless (Taylor’s Version)	25	Hey, I knew I’d run into you somewhere It’s been a while, I didn’t mean to stare I heard she’s nothin’ like me I’m sure she’ll make you happy But don’t you Don’t you smile at me and ask me how I’ve been Don’t you say you’ve missed me if you don’t want me again You don’t know how much I feel, I love you still So why don’t you, don’t you? Ah, ah, ah, ah Ah, ah, ah … ***** This Lyrics is NOT for Commercial use ***** (1409624132591)	208608
Better Man (Taylor’s Version) (From The Vault)	Red (Taylor’s Version)	22	I know I’m probably better off on my own Than lovin’ a man who didn’t know What he had when he had it And I see the permanent damage you did to me Never again, I just wish I could forget when it was magic I wish it wasn’t 4 a.m., standing in the mirror Saying to myself, you know you had to do it I know the bravest thing I ever did was run Sometimes, in the middle of the night, I can feel you again But I just miss you, and I just wish you were a better man But I know why we had to say goodbye like the back of my hand But I just miss you, and I just wish you were a better man A better man I know I’m probably better off all alone Than needing a man who could change his mind at any given minute And it was always on your terms I waited on every careless word Hoping they might turn sweet again … ***** This Lyrics is NOT for Commercial use ***** (1409624132591)	297013
A Perfectly Good Heart	Taylor Swift	14	Why would you wanna break a perfectly good heart? Why would you wanna take our love and tear it all apart, now? Why would you wanna make the very first scar? Why would you wanna break a perfectly good heart? Maybe I should’ve seen the signs, should’ve read the writing on the wall And realized by the distance in your eyes that I would be the one to fall No matter what you say, I still can’t believe That you would walk away … ***** This Lyrics is NOT for Commercial use ***** (1409624132591)	220146
Out Of The Woods (Taylor’s Version)	1989 (Taylor’s Version) [Deluxe]	4	Looking at it now It all seems so simple We were lying on your couch I remember You took a Polaroid of us Then discovered (then discovered) The rest of the world was black and white But we were in screaming color And I remember thinking Are we out of the woods yet? Are we out of the woods yet? Are we out of the woods yet? Are we out of the woods? Are we in the clear yet? Are we in the clear yet? Are we in the clear yet, in the clear yet? Good Are we out of the woods yet? Are we out of the woods yet? Are we out of the woods yet? Are we out of the woods? Are we in the clear yet? Are we in the clear yet? Are we in the clear yet, in the clear yet? Good Are we out of the woods? Looking at it now Last December (last December) We were built to fall apart And fall back together (back together) … ***** This Lyrics is NOT for Commercial use ***** (1409624132591)	235800
Daylight	Lover	18	My love was as cruel as the cities I lived in Everyone looked worse in the light There are so many lines that I’ve crossed unforgiven I’ll tell you the truth, but never goodbye I don’t wanna look at anything else now that I saw you I don’t wanna think of anything else now that I thought of you I’ve been sleeping so long in a 20-year dark night And now I see daylight, I only see daylight Luck of the draw only draws the unlucky And so I became the butt of the joke I wounded the good and I trusted the wicked Clearing the air, I breathed in the smoke Maybe you ran with the wolves and refused to settle down Maybe I’ve stormed out of every single room in this town Threw out our cloaks and our daggers because it’s morning now It’s brighter now, now I don’t wanna look at anything else now that I saw you (I can never look away) … ***** This Lyrics is NOT for Commercial use ***** (1409624132591)	293453
Don’t Blame Me	reputation	4	Don’t blame me, love made me crazy If it doesn’t, you ain’t doin’ it right Lord, save me, my drug is my baby I’ll be usin’ for the rest of my life I’ve been breakin’ hearts a long time And, toyin’ with them older guys Just playthings for me to use Something happened for the first time In the darkest little paradise Shakin’, pacin’, I just need you For you, I would cross the line I would waste my time I would lose my mind They say, she’s gone too far this time Don’t blame me, love made me crazy If it doesn’t, you ain’t doin’ it right Lord, save me, my drug is my baby I’ll be usin’ for the rest of my life Don’t blame me, love made me crazy If it doesn’t, you ain’t doin’ it right Oh, Lord, save me, my drug is my baby I’ll be usin’ for the rest of my life … ***** This Lyrics is NOT for Commercial use ***** (1409624132591)	236413
The Last Time (feat. Gary Lightbody of Snow Patrol) (Taylor’s Version)	Red (Taylor’s Version)	10	I find myself at your door Just like all those times before I’m not sure how I got there All roads, they lead me here I imagine you are home In your room, all alone And you open your eyes into mine And everything feels better And right before your eyes I’m breaking No past, no reasons why Just you and me This is the last time I’m asking you this Put my name at the top of your list This is the last time I’m asking you why You break my heart in the blink of an eye, eye, eye You find yourself at my door And just like all those times before You wear your best apology But I was there to watch you leave And all the times I let you in … ***** This Lyrics is NOT for Commercial use ***** (1409624132591)	299080
Carolina - From The Motion Picture “Where The Crawdads Sing”	Carolina (From The Motion Picture “Where The Crawdads Sing”)	1	Oh, Carolina creeks running through my veins Lost, I was born; lonesome, I came Lonesome I’ll always stay Carolina knows why for years I roam Free as these birds, light as whispers Carolina knows And you didn’t see me here No, they never did see me here And she’s in my dreams Into the mist, into the clouds Don’t leave I make a fist, I make it count And there are places I will never-ever go, oh-oh-oh And things that only Carolina will ever know … ***** This Lyrics is NOT for Commercial use ***** (1409624132591)	264438
willow	evermore (deluxe version)	1	I’m like the water when your ship rolled in that night Rough on the surface, but you cut through like a knife And if it was an open-shut case I never would’ve known from that look on your face Lost in your current like a priceless wine The more that you say, the less I know Wherever you stray, I follow I’m begging for you to take my hand Wreck my plans, that’s my man Life was a willow, and it bent right to your wind Head on the pillow, I can feel you sneaking in As if you were a mythical thing Like you were a trophy or a champion ring But there was one prize I’d cheat to win The more that you say, the less I know Wherever you stray, I follow I’m begging for you to take my hand Wreck my plans, that’s my man … ***** This Lyrics is NOT for Commercial use ***** (1409624132591)	214706
Midnight Rain	Midnights (The Til Dawn Edition)	6	Rain, he wanted it comfortable I wanted that pain He wanted a bride I was making my own name Chasing that fame He stayed the same All of me changed like midnight My town was a wasteland Full of cages, full of fences Pageant queens and big pretenders But for some, it was paradise My boy was a montage A slow-motion, love potion Jumping off things in the ocean I broke his heart ’cause he was nice He was sunshine, I was midnight rain He wanted it comfortable I wanted that pain He wanted a bride … ***** This Lyrics is NOT for Commercial use ***** (1409624132591)	174782
Cruel Summer - Live from TS \| The Eras Tour	The Cruelest Summer	1	Oh, hi! Fever dream high in the quiet of the night You know that I caught it Bad, bad boy, shiny toy with a price You know that I bought it Killing me slow, out the window I’m always waiting for you to be waiting below Devils roll the dice, angels roll their eyes What doesn’t kill me makes me want you more And it’s new, the shape of your body It’s blue, the feeling I got And it’s ooh, whoa-oh It’s a cruel summer It’s cool, that’s what I tell ’em No rules in breakable heaven But ooh, whoa-oh It’s a cruel summer with you Hang your head low in the glow of the vending machine I’m not dying We say that we’ll just screw it up in these trying times We’re not trying So cut the headlights, summer’s a knife I’m always waiting for you just to cut to the bone … ***** This Lyrics is NOT for Commercial use ***** (1409624132591)	229566

Exploratory Data Analysis

Because our dataset was manually curated, there exists no missing data.

vis_miss(official_list)

Eras Visualizations

We explore Swift’s musical catalog through her different eras (a loose term for her albums, but including songs that were not included in the album but released around the same time).

ggplot(official_list %>% 
  mutate(era = ifelse("Taylor Swift" == official_list$album_name.y, "Self-titled", ifelse(grepl("Fearless", official_list$album_name.y), "Fearless", 
  ifelse(grepl("Speak Now", official_list$album_name.y), "Speak Now",  
  ifelse(grepl("Red", official_list$album_name.y) | grepl("Message In A Bottle", official_list$album_name.y) | grepl("All Too Well", official_list$album_name.y), "RED", 
  ifelse(grepl("1989", official_list$album_name.y) | grepl("Sweeter Than Fiction", official_list$album_name.y), "1989",
  ifelse(grepl("reputation", official_list$album_name.y), "reputation", 
  ifelse(grepl("Christmas Tree Farm", official_list$album_name.y) | grepl("Lover", official_list$album_name.y) | grepl("Americana", official_list$album_name.y) | grepl("The Cruelest Summer", official_list$album_name.y) | grepl("All Of The Girls You Loved Before", official_list$album_name.y), "Lover", 
  ifelse(grepl("folklore", official_list$album_name.y) | grepl("Carolina", official_list$album_name.y) | grepl("the lakes", official_list$album_name.y) | grepl("Renegade", official_list$album_name.y), "folklore", 
  ifelse(grepl("evermore", official_list$album_name.y) | grepl("The Joker And The Queen", official_list$album_name.y), "evermore",  
         ifelse(grepl("Midnights", official_list$album_name.y) | grepl("You're Losing Me", official_list$album_name.y) | grepl("Anti-Hero", official_list$album_name.y), "Midnights", NA))))))))))) %>% 
  mutate(era = factor(era, levels = c("Self-titled", "Fearless", "Speak Now", "RED", "1989", "reputation", "Lover", "folklore", "evermore", "Midnights"))) %>% 
  arrange(era), aes(x= era, fill = era)) +
  geom_bar(stat = "count") + 
  theme_linedraw() + 
  theme(axis.ticks.x = element_blank(), axis.ticks.y = element_blank(), axis.text.x = element_text(angle = 90, vjust = 0.5, hjust = 0.5), legend.position = "none") + 
  scale_y_continuous(expand=c(0,0)) +
  scale_x_discrete(labels = c("Self-titled", "Fearless", "Speak Now", "RED", "1989", "reputation", "Lover", "folklore", "evermore", "Midnights")) +
  scale_fill_manual(values = c("#b9d2b5", "#f4cb8d", "#d1b2d2", "#823549", "#b5e9f6", "#847e80", "#f9b2d0", "#cfcac6", "#c8ae95", "#434961")) +
  labs(title = "Taylor Swift Songs per Era", x = "Era",
       y = element_blank()) +
  coord_flip()

With this visualization, we can see that her RED era featured the most songs. An album with a plethora of songs- including All Too Well about heartbreak or 22 about embracing independence and change, each individual era is not defined by a single theme or topic. This helps us see the challenge of the task we’re taking on- every era has a mix of different things to sing along to.

Text Analysis on Lyrics

Let’s get to the lyrics!

I create a lyricFreq() function in order to determine the most frequent words in each songs.

Because each lyric from Musixmatch ends with a disclaimer message, I account for the string until the … character. Following, I filter out symbols and punctuation points. To begin our text analysis, I account for stopwords and stemming. Using the SnowballC package, I am able to use the predefined dictionary of stopwords and stemmed words. However, since some of Swift’s songs are dependent on ‘stem words,’ I exclude a selected few: ‘me/mine’, ‘our’, ‘your’, etc.

ts_stopwords <- as_tibble(stopwords::stopwords(source = 'snowball')) %>%
  filter(!(value %in% c("me", "our", "ours", "your", "did", "would", "could", "should", "before", "after", "between", "against", "above", "below", "up", "down", "again", "why", "only", "same", "too", "will", "myself")))

lyricFreq <- function(index) {
  # this is to find when the lyrics end so we don't have to deal with the copyright message
  end <- which(strsplit(official_list$lyrics_body[index], split = ' ')[[1]] == "...")
  # to split up the lyric text by word 
  rv <- strsplit(tolower(official_list$lyrics_body[index]), split = ' ')[[1]][1:(end-1)] 
  rv <- gsub('[,()"!?]', '', rv)

  # this is a data frame of all the frequent words (in order)
  #return(count(rv)[order(count(rv)$freq, decreasing = T),])
  return(as_tibble(rv) %>%
  filter(!(value %in% as.vector(ts_stopwords)$value)) %>%
  mutate(stem = wordStem(value)) %>%
  count(stem) %>%
  arrange(desc(n)))
}

We run the lyrics and analysis for selected songs:

Say Don’t Go (Taylor’s Version), 1989 (Taylor’s Version)

official_list$lyrics_body[153]

[1] "I've known it from the very start We're a shot in the darkest dark Oh no, oh no, I'm unarmed The waiting is a sadness Fading into madness Oh no, oh no, it won't stop  I'm standin' on a tightrope alone I hold my breath a little bit longer Halfway out the door, but it won't close I'm holdin' out hope for you to say, \"Don't go\" I would stay forever if you say, \"Don't go\"  Why'd you have to lead me on? Why'd you have to twist the knife? Walk away and leave me bleedin', bleedin' Why'd you whisper in the dark Just to leave me in the night? Now your silence has me screamin', screamin'  (Say) say, \"(Don't) don't (go) go\" I would stay forever if you (say) say, \"(Don't) don't (go) go\" (Say, say, say, say)  ...  ******* This Lyrics is NOT for Commercial use ******* (1409624132591)"

knitr::kable(lyricFreq(153)) %>% 
  kable_styling(full_width = T) %>% 
  scroll_box(width = "30%", height = "200px")

stem	n
sai	10
go	6
	4
me	4
oh	4
why’d	3
bleedin’	2
dark	2
forev	2
leav	2
screamin’	2
stai	2
would	2
alon	1
awai	1
bit	1
breath	1
close	1
darkest	1
door	1
fade	1
halfwai	1
hold	1
holdin’	1
hope	1
just	1
knife	1
known	1
lead	1
littl	1
longer	1
mad	1
night	1
now	1
sad	1
shot	1
silenc	1
standin’	1
start	1
stop	1
tightrop	1
twist	1
unarm	1
wait	1
walk	1
whisper	1
your	1

wordcloud(words = lyricFreq(153)$stem, 
          freq = lyricFreq(153)$n, 
          min.freq = 1,
          max.words = 200,
          colors = brewer.pal(8, "Dark2"))

You’re Losing Me, Midnights (The Late Night Edition)

official_list$lyrics_body[218]

[1] "All this time I was wasting hoping you would come around I've been giving out chances every time and all you do is let me down And it's taken me this long, baby, but I've figured you out And you're thinking we'll be fine again but not this time around  You don't have to call anymore I won't pick up the phone This is the last straw Don't wanna hurt anymore And you can tell me that you're sorry but I don't believe you, baby, like I did before You're not sorry No, no, oh-oh  You're looking so innocent, I might believe you if I didn't know Could've loved you all my life if you hadn't left me waiting in the cold And you got your share of secrets and I'm tired of being last to know, oh ...  ******* This Lyrics is NOT for Commercial use ******* (1409624132591)"

knitr::kable(lyricFreq(218)) %>% 
  kable_styling(full_width = T) %>% 
  scroll_box(width = "30%", height = "200px")

stem	n
me	4
time	3
	2
anymor	2
around	2
babi	2
believ	2
know	2
last	2
sorri	2
again	1
befor	1
call	1
can	1
chanc	1
cold	1
come	1
could’v	1
did	1
down	1
everi	1
figur	1
fine	1
give	1
got	1
hope	1
hurt	1
innoc	1
left	1
let	1
life	1
like	1
long	1
look	1
love	1
might	1
oh	1
oh-oh	1
phone	1
pick	1
secret	1
share	1
straw	1
taken	1
tell	1
think	1
tire	1
up	1
wait	1
wanna	1
wast	1
would	1
your	1

wordcloud(words = lyricFreq(218)$stem, 
          freq = lyricFreq(218)$n, 
          min.freq = 2,
          max.words = 200,
          colors = brewer.pal(3, "Dark2"))

All Too Well (10 Minute Version) (Taylor’s Version) (From The Vault), RED (Taylor’s Version)

official_list$lyrics_body[9]

[1] "I walked through the door with you The air was cold But something 'bout it felt like home somehow And I, left my scarf there at your sister's house And you've still got it in your drawer even now  Oh, your sweet disposition And my wide-eyed gaze We're singing in the car, getting lost upstate Autumn leaves falling down like pieces into place And I can picture it after all these days  And I know it's long gone and that magic's not here no more And I might be okay, but I'm not fine at all  'Cause there we are again on that little town street You almost ran the red 'cause you were looking over at me Wind in my hair, I was there I remember it all too well  Photo album on the counter Your cheeks were turning red You used to be a little kid with glasses in a twin-sized bed And your mother's telling stories 'bout you on the tee-ball team You told me 'bout your past thinking your future was me  And you were tossing me the car keys \"F- the patriarchy\" keychain on the ground We were always skipping town And I was thinking on the drive down Any time now, he's gonna say it's love You never called it what it was  'Til we were dead and gone and buried Check the pulse and come back swearing it's the same After three months in the grave And then you wondered where it went to as I reached for you But all I felt was shame, and you held my lifeless frame  And I know it's long gone, and there was nothing else I could do And I forget about you long enough to forget why I needed to  'Cause there we are again in the middle of the night ...  ******* This Lyrics is NOT for Commercial use ******* (1409624132591)"

knitr::kable(lyricFreq(9)) %>% 
  kable_styling(full_width = T) %>% 
  scroll_box(width = "30%", height = "200px")

stem	n
	8
your	7
me	4
’bout	3
’caus	3
gone	3
long	3
after	2
again	2
car	2
down	2
felt	2
forget	2
know	2
like	2
littl	2
now	2
red	2
think	2
town	2
’til	1
air	1
album	1
almost	1
alwai	1
autumn	1
back	1
bed	1
buri	1
call	1
can	1
check	1
cheek	1
cold	1
come	1
could	1
counter	1
dai	1
dead	1
disposit	1
door	1
drawer	1
drive	1
els	1
enough	1
even	1
f-	1
fall	1
fine	1
frame	1
futur	1
gaze	1
get	1
glass	1
gonna	1
got	1
grave	1
ground	1
hair	1
held	1
home	1
hous	1
kei	1
keychain	1
kid	1
leav	1
left	1
lifeless	1
look	1
lost	1
love	1
magic’	1
middl	1
might	1
month	1
mother’	1
need	1
never	1
night	1
noth	1
oh	1
okai	1
past	1
patriarchi	1
photo	1
pictur	1
piec	1
place	1
puls	1
ran	1
reach	1
rememb	1
sai	1
same	1
scarf	1
shame	1
sing	1
sister’	1
skip	1
somehow	1
someth	1
still	1
stori	1
street	1
swear	1
sweet	1
team	1
tee-bal	1
tell	1
three	1
time	1
told	1
too	1
toss	1
turn	1
twin-siz	1
upstat	1
us	1
walk	1
well	1
went	1
why	1
wide-ei	1
wind	1
wonder	1

wordcloud(words = lyricFreq(9)$stem, 
          freq = lyricFreq(9)$n, 
          min.freq = 1,
          max.words = 200,
          colors = brewer.pal(8, "Set1"))

This function helps me identify what keywords are commonly used in Swift’s songs, in order for me to produce a more precise dictionary.

Creating a Dictionary

I create a dictionary for the different themes of Swift’s songs: love, heartache/pain, growing up/change, and self-reflection/reputation. By listening to her music, I generalized her tracks into these common themes. For instance, Enchanted (Taylor’s Version) which is about a new crush would be categorized as a love song. On the other hand, Anti-Hero which details how she views herself in response to her publicity, would be about reflection/reputation. Noticing common words that she would sing in her songs as well as using our EDAs, I curated the dictionary terms with her frequently used jargon.

To create my dictionary for the love theme, I referenced song lyrics such as: “Romeo, take me somewhere we can be alone / I’ll be waiting, all that’s left to do is run” from Love Story (Taylor’s Version), “If you could see that I’m the one who understands you / Been here all along, so why can’t you see?” from You Belong With Me (Taylor’s Version), “This night is sparkling, don’t you let it go / I’m wonderstruck, blushing all the way home” from Enchanted (Taylor’s Version), “Drop everything now, meet me in the pouring rain / Kiss me on the sidewalk, take away the pain” from Sparks Fly (Taylor’s Version), “He’s so tall and handsome as hell / He’s so bad, but he does it so well” from Wildest Dreams (Taylor’s Version), and “One night, he wakes, strange look on his face / Pauses, then says, ‘You’re my best friend.’ / And you knew what it was, he is in love” from You Are In Love (Taylor’s Version).

To create my dictionary for pain, I reference song lyrics such as: “So this is me swallowing my pride, standing in front of you, saying I’m sorry for that night” from Back to December (Taylor’s Version), “So I’ll watch your life in pictures like I used to watch you sleep / And I feel you forget me like I used to feel you breathe” from Last Kiss (Taylor’s Version), “If you tasted poison, you could’ve / Spit me out at the first chance / If I was some paint, did it splatter / On a promising grown man?” from Would’ve, Could’ve, Should’ve, and “Because I dropped your hand while dancing / Left you out there standing / Crestfallen on the landing” from champagne problems.

To create my dictionary for change, I referenced song lyrics such as: “Like any great love, it keeps you guessing / Like any real love, it’s ever-changing” from Welcome to New York (Taylor’s Version), “I’ve been the archer / I’ve been the prey / Who could ever leave me, darling?” from The Archer, or “Back then I swore I was gonna marry him someday / But I realized some bigger dreams of mine” from Fifteen (Taylor’s Version).

To create my dictionary for reflection, I referenced song lyrics such as: “I’m so sick of running as fast as I can / Wondering if I’d get there quicker / If I was a man” from The Man or “Everybody’s waiting for you to break down / Everybody’s watching to see the fallout” from Eyes Open (Taylor’s Version).

myDict <- dictionary(list(love = c('love*', 'girl*', 'forev*', 'light*', 'first', 'summer', 'time', 'friend', 'eye', 'see', 'bab*', 'romeo', 'run', 'one', 'spark*', 'wonder*', 'enchant*', 'handsom*', 'best', 'daydream', 'treacherous', 'fearless', 'style', 'grin*', 'electr*', 'flawless', 'hand', 'craz*', 'favorit*', 'lip*', 'blind', 'book', 'pac*', 'town', 'sunshin*', 'dress', 'breez', 'dream', 'fairytal*', 'save', 'tall', 'somedai', 'smile', 'laugh', 'togeth', 'spark', 'belong', 'girlfriend', 'red', 'invisib*', 'begin', 'wild*', 'magic*', 'timeless', 'boy*', 'wildest', 'heaven', 'sunset', 'arms', 'different', 'silence', 'sweet', 'skin', 'high', 'angel', 'glow', 'shape', 'bad', 'ring'),
                          pain = c('remember', 'never', 'lost', 'lose', 'losin*', 'forget', 'dead', 'cruel', 'fall', 'right', 'wait', 'sorry', 'swallow', 'blurry', 'worth', 'breathe', 'gray', 'shade', 'away', 'sad', 'tragic', 'tear*', 'richochet', 'last', 'stream*', 'mess*', 'cri*', 'paralyz*', 'kill*', 'exile', 'trouble', 'problem*', 'final', 'champagn', 'crestfallen', 'worst', 'infidel', 'broken', 'regret', 'wound', 'ghost', 'poison', 'scare', 'weapon', 'pain', 'reveng', 'fake', "ain't", "should'v", 'perfectli', 'apart', 'sign', 'complic', 'break', 'broke', 'empti', 'speak', 'haunt*', 'last', 'breakd*', 'think', 'mcgraw', 'fade', 'shame', 'toler*', 'better', 'desper*', "could'v", "would'v", 'apologi*', 'door', 'fight', 'god', 'screaming', 'woods', 'sad', 'without', 'tire*', 'called', 'car', 'wash', 'stupid', 'anymor', 'storm'),
                          change = c('face', 'year*', 'change', 'happi', 'free', 'confus*', 'same', 'revolution', 'chance', 'day*', 'grow', 'grew', 'young', 'littl*', 'guess*', 'live', 'pretend', 'york', 'welcom', 'waitin*', 'danc*', 'forevermor', 'loud', 'ash', 'best', 'fireplac*', 'learn', 'sprinkler', 'know', 'year', 'new', 'proud*', '22', 'alright', 'crowd', 'creek', 'pennsylvania', 'innoc*', 'balanc*', 'bigger', 'combat', 'easi*', 'archer', 'win', 'darl*', 'shake', 'peopl*', 'world', 'city'),
                          reflection = c('hero', 'reputation', 'dynasti', 'power', 'play*', 'man', 'rude', 'mirrorball', 'version', 'believe', 'natural', 'try', 'empir', 'bridg*', 'castle*', 'watch', 'everybod*', 'fallout', 'kill', 'proof', 'woman', 'women', 'gold', 'american*', 'everybod*', 'depress*', 'anti-hero', 'game', 'underlin*', 'cynic*', 'hunter', 'peak', 'nothin*', 'karma', 'street', 'vice', 'wise*')))

Preprocessing our Data

Now, it’s time to get our data ready for machine learning. To preprocess our data, we will:

Tokenize our text- divide the different tests into individuals words
Remove stop words- remove common words that do not imply meaningful analysis
Stem words- reduce a word to its root format (to recognize common words but used in various tenses)

Feature Extraction- Text Mining

Using the quanteda package, I transform the data frame (i.e. a corpus object- the data structure for text data before tokenization) of lyrics into a document-feature matrix (DFM).

A DFM is a matrix in which the elements are the counts of the features in the documents. In our project, our features columns represents each individual string/word that Swift uses in all her 219 songs, and each row represents a different doc, i.e each song.

I use tokens() from the quenteda package to break down the song lyrics into individual words (i.e. tokens). This tokenization process is a fundamental step in the Natural Language Processing (NLP). I also remove the stopwords that we crafted above, change all of the words to lowercase for consistency, and remove Musixmatch’s closing copyright tag.

ts_dtm2 <- tokens(corpus(official_list$lyrics_body)) %>% 
  tokens_remove(pattern = ts_stopwords) %>% 
  tokens_tolower() %>% 
  tokens_remove(pattern = c('lyrics', 'commercial', 'use', '1409624132591'))

ts_dtm2 <- ts_dtm2 %>% dfm()

So, this is what the DFM looks like!

doc_id	he	was	a	time	that	i	him	wonder	had	and	but	if	he’s	,	can	be	’	for	too	don’t	at	all	my	like	so	see	keep	this	is	go	me	never	alone	low	no	to	know	in	the	of	night	you	should	things	we	do	baby	(	)	i’m	gonna	with	i’ll	it	?	.		not	were	gone	would	have	it’s	up	glass	when	there	by	call	even	looks	now	always	right	bad	young	what	got	head	love	think	on	knows	price	won’t	might	well	us	perfect	dress	make	our	fall	yeah	oh	about	feeling	everything	will	bet	want	just	nights	place	why	perfectly	heart	apart	maybe	should’ve	wall	your	eyes	can’t	away	cause	out	through	rain	way	that’s	life	goes	girl	you’re	blue	back	face	blame	lose	phone	around	mine	into	ooh	i’ve	every	home	getting	leaves	pieces	after	days	long	here	more	okay	fine	again	little	told	”	changed	stay	let	been	get	end	day	feel	moment	wine	floor	ain’t	how’s	tell	mind	die	turn	dark	em	cut	did	shiny	high	song	funny	their	tonight	whole	meet	boy	goodbye	needing	change	his	summer	she	she’s	says	side	single	sky	live	without	heaven	game	new	dying	killing	simple	smile	lipstick	friend	once	hang	which	finally	glow	takes	shape	laugh	window	wondering	bar	casually	waiting	fever	dream	quiet	caught	toy	bought	slow	below	devils	roll	dice	angels	doesn’t	kill	makes	body	whoa-oh	cruel	cool	rules	breakable	vending	machine	lived	everyone	looked	revolved	counting	footsteps	praying	mother	accused	losing	swore	paint	chess	version	stopped	picking	dear	john	messed	cried	known	blind	optimism	misery	since	met	each	hello	yours	breaking	sleep	tuesday	eye	her	gold	anyone	beautiful	infidelity	different	feelings	against	talks	dinner	mouth	prove	crime	fake	mr	until	wonderful	revolves	ms	cry	este’s	losin	husband’s	actin	smells	merlot	jewelry	joint	account	doubt	lettin	drew	magician	disappear	cage	hostage	trippin	trip-trippin	breakdown	illusionist	everythin
text129	0	2	0	1	2	3	0	0	1	2	2	0	0	4	0	1	0	2	0	0	2	2	1	0	0	2	0	1	1	1	5	3	1	0	0	5	1	2	4	1	0	4	0	0	0	0	1	1	1	0	0	0	0	1	1	18	14	1	0	0	1	0	1	1	0	1	0	1	0	0	0	0	1	3	0	0	0	1	0	0	0	0	0	0	0	0	0	0	1	0	0	0	1	0	0	0	0	3	1	0	0	1	0	1	1	2	2	1	0	0	0	3	0	0	1	0	0	0	0	0	0	1	0	0	2	0	0	1	0	0	0	1	1	0	0	2	1	0	0	1	0	1	0	0	1	0	1	2	0	0	2	30	0	1	0	2	1	1	2	0	0	0	0	0	1	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	1	0	1	0	0	0	0	0	1	1	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	1	0	0	0	0	0	1	1	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	1	0	0	0	0	0	0	0	1	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	1	1	0	1	2	0	1	0	0	1	0	0	0	0	0	0	0	0	0	0	0	0	0	0	14	1	1	1	1	1	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0
text46	0	0	3	0	2	6	0	0	0	2	1	0	0	19	0	1	1	2	0	0	0	0	0	0	0	0	0	1	1	0	3	0	0	1	1	1	2	3	8	3	1	5	0	0	0	0	0	5	5	2	0	2	0	5	0	3	14	2	0	0	0	0	6	0	0	0	0	0	0	0	0	0	1	3	2	0	2	1	1	0	0	0	0	1	0	0	0	0	0	0	0	0	0	7	3	0	1	0	0	0	4	0	0	0	0	0	0	0	0	0	0	2	1	0	0	0	1	0	0	0	1	0	0	0	3	1	0	0	0	0	0	0	0	0	2	1	0	0	0	0	0	0	0	0	0	1	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	1	0	0	0	0	1	0	0	1	1	0	0	1	0	0	0	1	0	0	0	0	2	0	0	0	0	0	0	0	0	1	0	1	1	1	0	0	0	0	0	1	0	0	1	0	1	0	1	0	0	0	2	1	1	1	1	1	1	1	1	1	2	1	1	1	1	1	1	2	2	1	1	1	1	1	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0
text50	0	2	1	0	1	8	0	0	0	6	2	0	0	6	0	1	0	1	1	1	0	1	5	0	0	1	0	2	2	1	3	0	0	0	0	4	1	2	7	2	0	6	0	0	0	0	0	1	1	0	0	1	0	2	1	3	14	1	1	1	0	0	1	1	0	1	0	0	0	0	0	1	0	0	0	1	0	0	0	0	1	1	0	0	1	1	2	0	0	1	0	0	1	0	0	0	0	0	0	0	0	0	1	0	1	0	0	0	1	1	0	1	0	0	0	0	0	1	1	1	0	0	0	1	1	1	1	0	1	0	1	1	0	0	0	0	1	1	0	0	0	0	1	1	0	0	0	1	1	0	0	0	1	0	1	0	1	0	1	0	0	0	1	0	0	0	1	0	1	0	0	0	0	0	0	1	0	0	1	1	0	0	0	0	0	0	0	0	0	0	0	0	1	0	0	0	1	0	0	0	0	0	0	0	1	0	1	0	0	0	0	0	0	1	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	1	0	0	0	1	0	0	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	1	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0
text134	5	0	2	0	2	9	1	0	0	2	4	0	0	10	0	0	3	2	0	0	0	0	2	1	0	0	0	1	1	0	0	0	0	0	7	0	0	0	1	2	1	0	0	0	1	0	0	1	1	1	1	0	0	9	0	3	14	1	0	0	0	0	0	2	1	0	1	0	1	0	0	0	0	0	0	0	0	0	0	0	4	2	0	0	0	0	0	0	0	0	0	1	0	0	0	0	0	0	0	0	0	3	0	0	0	0	0	0	0	0	0	0	0	3	0	0	1	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	1	0	0	0	1	0	0	0	0	0	0	0	0	0	0	0	0	0	0	5	0	0	0	2	0	0	1	0	0	1	0	4	0	0	0	1	0	0	0	0	5	0	0	0	0	0	0	0	1	0	0	0	0	1	0	2	0	2	0	0	0	0	0	0	0	0	0	0	0	0	0	1	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	1	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	1	1	0	1	0	0	0	1	1	0	0	0	1	1	3	1	0	0	1	0	0	0	0	2	1	1	1	1	1	1	1	1	1	1	0	0	0	0	0	0	0	0	0	0
text159	0	0	4	0	0	2	0	0	0	3	2	0	0	7	1	0	4	1	0	0	0	5	2	0	3	1	1	1	1	0	3	0	1	0	0	3	3	3	3	0	0	8	0	1	2	1	1	1	1	3	0	1	0	3	0	3	14	2	0	1	0	0	1	1	0	2	0	0	0	0	0	0	0	1	2	0	1	0	0	0	0	5	0	0	0	0	0	2	0	0	1	0	1	0	0	0	0	0	0	0	0	0	0	1	0	0	0	0	0	0	1	2	4	0	0	2	0	0	0	0	0	0	2	1	1	0	1	1	0	1	0	0	0	2	0	0	0	0	1	0	2	0	0	0	0	0	0	0	0	1	0	0	0	0	0	0	1	0	0	1	1	0	0	0	0	0	0	0	0	1	0	1	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	1	0	1	0	0	0	0	0	0	0	0	0	0	0	1	0	0	0	0	0	1	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	1	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	1	0	0	2	0	0	0	0	0	1	0	0	0	0	1	1	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	1	1	1	1	1	1	1	1	0
text173	4	0	1	0	4	6	0	1	0	2	1	1	3	0	0	1	3	1	0	0	1	0	0	0	3	2	0	1	1	0	3	0	0	0	0	2	0	1	0	0	0	0	1	0	1	0	0	1	1	1	0	1	1	1	0	3	14	1	0	0	0	1	1	0	0	1	0	0	0	1	1	0	0	1	0	0	0	2	0	1	0	0	1	0	1	0	0	0	0	0	0	0	0	0	0	1	0	0	0	1	1	1	0	0	0	0	0	0	0	0	0	0	0	1	0	1	0	0	0	0	0	0	0	1	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	1	0	0	0	0	0	0	1	0	0	0	0	2	1	0	0	0	1	1	0	0	0	0	0	0	1	0	0	0	0	0	1	0	0	0	1	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	1	1	0	0	0	0	2	0	0	0	0	1	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	2	0	0	0	0	0	0	0	0	2

It won’t provide us much information until we use the dfm_lookup() function to cross-reference the words with my manually created Taylor Swift dictionary for the common themes.

ts_dfm <- dfm_lookup(ts_dtm2, dictionary = myDict)

ts_dfm <- ts_dfm %>% as.data.frame() %>% 
  cbind(official_list$track_name) %>% 
  select(-c('doc_id')) %>% 
  select(c(5, 1:4))

Our useful document feature matrix:

official_list$track_name	love	pain	change	reflection
…Ready For It?	16	7	2	3
’tis the damn season	3	3	3	0
“Slut!” (Taylor’s Version) (From The Vault)	5	3	1	0
22 (Taylor’s Version)	6	2	11	0
A Perfectly Good Heart	3	6	0	1
A Place in this World	3	0	4	0
Afterglow	3	4	1	0
All Of The Girls You Loved Before	6	2	0	1
All Too Well (10 Minute Version) (Taylor’s Version) (From The Vault)	9	11	6	2
All Too Well (10 Minute Version) (The Short Film)	9	12	7	2
All Too Well (Sad Girl Autumn Version) - Recorded at Long Pond Studios	10	11	7	2
All Too Well (Taylor’s Version)	4	4	3	1
All You Had To Do Was Stay (Taylor’s Version)	2	2	4	1
Anti-Hero	4	7	2	6
Anti-Hero (feat. Bleachers)	3	8	2	6
august	5	12	0	0
Babe (Taylor’s Version) (From The Vault)	12	9	0	0
Back To December (Taylor’s Version)	5	3	2	1
Bad Blood (feat. Kendrick Lamar) (Taylor’s Version)	12	9	2	0
Bad Blood (Taylor’s Version)	13	8	2	0
Begin Again (Taylor’s Version)	1	3	2	0
Bejeweled	9	4	2	2
Better Man (Taylor’s Version) (From The Vault)	5	7	7	5
Better Than Revenge (Taylor’s Version)	2	6	3	2
betty	5	3	3	2
Bigger Than The Whole Sky	1	3	2	0
Blank Space (Taylor’s Version)	11	4	5	4
Breathe (feat. Colbie Caillat) (Taylor’s Version)	6	8	6	1
Bye Bye Baby (Taylor’s Version) (From The Vault)	7	4	1	2
Call It What You Want	8	6	7	3
cardigan	10	2	8	3
Carolina - From The Motion Picture “Where The Crawdads Sing”	3	2	3	0
Castles Crumbling (feat. Hayley Williams) (Taylor’s Version) (From The Vault)	1	0	3	3
champagne problems	2	3	1	0
Change (Taylor’s Version)	3	3	5	1
Christmas Tree Farm	7	0	3	0
Christmas Tree Farm (Old Timey Version)	7	0	3	0
Clean (Taylor’s Version)	1	5	0	1
closure	2	5	3	0
Cold As You	1	6	3	0
Come Back…Be Here (Taylor’s Version)	1	3	5	1
Come In With The Rain (Taylor’s Version)	1	3	3	1
coney island (feat. The National)	3	4	0	0
Cornelia Street	3	11	1	5
cowboy like me	8	3	6	1
Cruel Summer	10	7	5	1
Cruel Summer - Live from TS \| The Eras Tour	10	4	6	1
Dancing With Our Hands Tied	14	1	13	1
Daylight	5	5	2	0
Dear John (Taylor’s Version)	5	5	4	2
Dear Reader	2	5	0	0
Death By A Thousand Cuts	5	2	2	1
Delicate	5	6	4	0
Don’t Blame Me	18	7	1	1
Don’t You (Taylor’s Version) (From The Vault)	3	0	1	1
dorothea	2	3	4	1
Dress	10	1	4	1
Electric Touch (feat. Fall Out Boy) (Taylor’s Version) (From The Vault)	10	3	1	0
Enchanted (Taylor’s Version)	7	1	2	1
End Game	10	2	2	8
epiphany	0	5	0	2
evermore (feat. Bon Iver)	3	5	1	0
Everything Has Changed (feat. Ed Sheeran) (Taylor’s Version)	1	10	18	0
exile (feat. Bon Iver)	6	4	1	0
Eyes Open (Taylor’s Version)	1	2	3	7
False God	8	8	3	0
Fearless (Taylor’s Version)	3	3	4	1
Fifteen (Taylor’s Version)	6	1	6	4
Foolish One (Taylor’s Version) (From The Vault)	6	5	7	0
Forever & Always (Taylor’s Version)	8	4	0	2
Forever Winter (Taylor’s Version) (From The Vault)	7	5	4	2
Getaway Car	6	15	2	2
Girl At Home (Taylor’s Version)	7	2	1	7
Glitch	2	3	1	0
gold rush	3	0	2	6
Gorgeous	2	1	4	0
happiness	4	3	4	0
Haunted (Taylor’s Version)	2	5	1	0
Hey Stephen (Taylor’s Version)	7	0	3	0
High Infidelity	4	5	8	1
Hits Different	10	4	2	3
hoax	2	4	1	1
Holy Ground (Taylor’s Version)	6	4	7	0
How You Get The Girl (Taylor’s Version)	7	6	0	0
I Almost Do (Taylor’s Version)	5	2	2	0
I Bet You Think About Me (feat. Chris Stapleton) (Taylor’s Version) (From The Vault)	3	4	3	1
I Can See You (Taylor’s Version) (From The Vault)	6	3	1	0
I Did Something Bad	7	4	1	4
I Forgot That You Existed	5	3	2	1
I Knew You Were Trouble (Taylor’s Version)	1	12	2	0
I Know Places (Taylor’s Version)	7	2	2	1
I Think He Knows	3	6	1	1
I Wish You Would (Taylor’s Version)	2	15	3	1
I’m Only Me When I’m With You	2	3	3	0
If This Was A Movie (Taylor’s Version)	3	3	5	1
illicit affairs	2	0	1	0
Innocent (Taylor’s Version)	3	4	9	1
Invisible	7	3	1	0
invisible string	7	2	2	0
Is It Over Now? (Taylor’s Version) (From The Vault)	8	2	0	0
It’s Nice To Have A Friend	4	1	0	0
it’s time to go - bonus track	2	1	7	0
ivy	4	2	3	0
Jump Then Fall (Taylor’s Version)	8	6	2	1
Karma	4	1	2	6
Karma (feat. Ice Spice)	5	1	1	9
King Of My Heart	4	3	6	2
Labyrinth	4	8	1	0
Last Kiss (Taylor’s Version)	4	6	4	0
Lavender Haze	0	0	2	0
London Boy	12	0	7	1
Long Live (Taylor’s Version)	6	5	6	1
long story short	4	4	0	0
Look What You Made Me Do	4	2	1	2
Love Story (Taylor’s Version)	11	1	5	0
Lover	3	0	1	0
mad woman	7	3	3	4
marjorie	0	16	2	2
Maroon	4	1	5	0
Mary’s Song (Oh My My My)	3	2	2	0
Mastermind	4	1	2	1
ME! (feat. Brendon Urie of Panic! At The Disco)	9	5	3	0
Mean (Taylor’s Version)	2	0	3	2
Message In A Bottle (Taylor’s Version) (From The Vault)	5	3	5	0
Midnight Rain	4	3	1	0
Mine (Taylor’s Version)	8	3	3	1
mirrorball	3	1	2	2
Miss Americana & The Heartbreak Prince	14	8	4	5
Mr. Perfectly Fine (Taylor’s Version) (From The Vault)	6	10	6	0
my tears ricochet	3	4	3	1
Never Grow Up (Taylor’s Version)	3	5	12	2
New Romantics (Taylor’s Version)	7	4	10	2
New Year’s Day	4	4	3	0
no body, no crime (feat. HAIM)	2	10	1	0
Nothing New (feat. Phoebe Bridgers) (Taylor’s Version) (From The Vault)	2	4	5	2
Now That We Don’t Talk (Taylor’s Version) (From The Vault)	2	2	8	1
Only The Young - Featured in Miss Americana	9	3	6	1
Our Song	3	3	1	3
Ours (Taylor’s Version)	6	1	3	1
Out Of The Woods (Taylor’s Version)	0	17	1	0
Paper Rings	7	3	4	2
Paris	3	0	1	0
peace	3	3	2	0
Picture To Burn	7	3	1	2
Question…?	5	3	1	0
Red (Taylor’s Version)	2	7	4	1
Renegade - Pop Version	5	0	1	1
right where you left me - bonus track	1	6	2	1
Ronan (Taylor’s Version)	5	6	5	1
Run (feat. Ed Sheeran) (Taylor’s Version) (From The Vault)	8	3	2	0
Sad Beautiful Tragic (Taylor’s Version)	4	4	3	0
Safe & Sound (feat. Joy Williams and John Paul White) (Taylor’s Version)	3	8	3	0
Say Don’t Go (Taylor’s Version) (From The Vault)	3	2	2	0
seven	5	0	3	1
Shake It Off (Taylor’s Version)	5	11	19	7
Should’ve Said No	2	1	1	0
Snow On The Beach (feat. Lana Del Rey)	4	0	0	0
Snow On The Beach (feat. More Lana Del Rey)	4	0	0	0
So It Goes…	7	4	5	1
Soon You’ll Get Better (feat. The Chicks)	6	4	4	0
Sparks Fly (Taylor’s Version)	10	3	1	0
Speak Now (Taylor’s Version)	6	5	2	1
Starlight (Taylor’s Version)	7	5	4	1
State Of Grace (Taylor’s Version)	3	5	3	0
Stay Beautiful	3	1	6	1
Stay Stay Stay (Taylor’s Version)	4	7	1	0
Style (Taylor’s Version)	8	3	2	1
Suburban Legends (Taylor’s Version) (From The Vault)	2	1	1	0
Superman (Taylor’s Version)	9	5	4	2
Superstar (Taylor’s Version)	10	3	1	2
Sweet Nothing	3	2	1	2
Sweeter Than Fiction - From “One Chance” Soundtrack	3	9	2	0
Teardrops on My Guitar - Pop Version	7	3	1	0
Tell Me Why (Taylor’s Version)	5	4	4	0
That’s When (feat. Keith Urban) (Taylor’s Version) (From The Vault)	3	1	3	0
the 1	1	7	7	0
The Archer	0	5	6	0
The Best Day (Taylor’s Version)	4	3	9	1
The Great War	5	4	0	1
The Joker And The Queen (feat. Taylor Swift)	3	2	3	0
the lakes - bonus track	2	1	0	1
the lakes - original version	2	2	0	2
the last great american dynasty	1	3	4	1
The Last Time (feat. Gary Lightbody of Snow Patrol) (Taylor’s Version)	6	7	1	1
The Lucky One (Taylor’s Version)	9	0	3	2
The Man	2	0	0	6
The Moment I Knew (Taylor’s Version)	10	3	3	0
The Other Side Of The Door (Taylor’s Version)	2	6	2	0
The Outside	4	3	3	2
The Story Of Us (Taylor’s Version)	2	5	6	1
The Very First Night (Taylor’s Version) (From The Vault)	4	2	4	0
The Way I Loved You (Taylor’s Version)	3	5	0	0
this is me trying	1	1	4	0
This Is Why We Can’t Have Nice Things	0	3	2	0
This Love (Taylor’s Version)	8	2	3	0
Tied Together with a Smile	8	1	3	0
Tim McGraw	7	11	2	0
Timeless (Taylor’s Version) (From The Vault)	8	5	1	2
Today Was A Fairytale (Taylor’s Version)	12	2	0	0
tolerate it	2	4	2	5
Treacherous (Taylor’s Version)	5	2	0	0
Untouchable (Taylor’s Version)	2	1	4	0
Vigilante Shit	7	5	0	6
We Are Never Ever Getting Back Together (Taylor’s Version)	5	9	2	0
We Were Happy (Taylor’s Version) (From The Vault)	5	0	0	0
Welcome To New York (Taylor’s Version)	4	1	25	1
When Emma Falls in Love (Taylor’s Version) (From The Vault)	14	2	5	0
White Horse (Taylor’s Version)	7	5	2	0
Wildest Dreams (Taylor’s Version)	17	2	2	1
willow	4	3	4	2
Wonderland (Taylor’s Version)	10	6	1	1
Would’ve, Could’ve, Should’ve	4	10	3	1
You All Over Me (feat. Maren Morris) (Taylor’s Version) (From The Vault)	0	5	2	0
You Are In Love (Taylor’s Version)	17	0	0	3
You Belong With Me (Taylor’s Version)	7	1	2	0
You Need To Calm Down	0	4	2	1
You’re Losing Me (From The Vault)	6	8	2	0
You’re Not Sorry (Taylor’s Version)	6	5	4	2
You’re On Your Own, Kid	8	4	8	1

Taking a brief glimpse at our document feature matrix, we can notice that a good number of our songs have been accurately matched to their correct theme: …Ready For It, Sl%t (Taylor’s Version), Call It What You Want, Cruel Summer, Don’t Blame Me, Enchanted (Taylor’s Version) all have the highest count for love; We Are Never Ever Getting Back Together (Taylor’s Version), Would’ve, Could’ve, Should’ve, You’re Losing Me, right where you left me all have the highest count for pain; Change (Taylor’s Version), Never Grow Up (Taylor’s Version), New Romantics (Taylor’s Version), Welcome to New York (Taylor’s Version) have the highest count for change; and Eyes Open (Taylor’s Version) has the highest for reflection. However, there are songs that have not been accurately matched to their correct theme or tied between 2 themes. For instance, Look What You Made Me Do or Anti-Hero were not assigned to the reflection theme. Since we’re only interpreting the lyrics of the first 30% of the song, it could make sense for the song to fall into the love category, rather than the reflection theme, due to her use of love-related words to set up the song’s narrative. In addition, Cornelia Street is a love song about a relationship being so good that Swift is afraid it might end; this doubled context is then interpreted as a song about pain rather than love by our dictionary of key words.

Interestingly, songs that I presumed would suit one theme but ended up falling into another theme however, still made valid sense. For instance, Blank Space (Taylor’s Version) details Swift’s love’s life in response to media scrutiny, but she conveys the message through a lover’s perspective- which is why it was categorized in the love theme, unlike reflection as expected.

Building Models

To build our unsupervised learning models, I intend to fit the following algorithms: K-Means Clustering and Hierarchical (Agglomerative).

K- Means Clustering

The K-means algorithm is an unsupervised learning algorithm that organizes observations into groups based on similarities. In context of our project, our algorithm organizes songs into their thematic groups. The algorithm iteratively assigns each song to its nearest cluster centroid, and then recalculates the centroids based on newly formed clusters. Initially, each centroid is placed randomly and move themselves to the center of the observations that are closer to them. The algorithm is finished when the position or the groups don’t change anymore at each iteration.

Tuning Number of Centers

To perform K- Means clustering on our document feature matrix, I tune different numbers of groups to cluster by. To prepare this model, I do not tune my DFM since my count values are already all scaled the same way.

set.seed(13)
kmeans(ts_dfm[,-1], centers = 2)

K-means clustering with 2 clusters of sizes 136, 83

Cluster means:
      love     pain   change reflection
1 3.036765 3.845588 2.676471  0.9705882
2 8.686747 4.349398 3.963855  1.4698795

Clustering vector:
  [1] 2 1 1 2 1 1 1 1 2 2 2 1 1 1 1 1 2 1 2 2 1 2 2 1 1 1 2 2 2 2 2 1 1 1 1 2 2
 [38] 1 1 1 1 1 1 1 2 2 2 2 1 1 1 1 1 2 1 1 2 2 2 2 1 1 1 1 1 2 1 2 2 2 2 2 2 1
 [75] 1 1 1 1 2 1 2 1 2 2 1 1 1 2 1 1 2 1 1 1 1 1 1 2 2 2 1 1 1 2 1 1 1 1 1 1 2
[112] 2 1 1 2 1 2 1 1 1 1 2 1 1 1 2 1 2 2 1 1 2 1 1 1 1 2 1 1 1 2 1 1 2 1 1 1 1
[149] 1 2 1 1 1 1 2 1 1 1 2 2 2 1 2 1 1 1 2 1 2 2 1 1 2 1 1 1 1 1 1 1 1 1 1 1 2
[186] 1 2 1 1 1 1 1 1 1 2 2 2 2 2 1 1 1 2 1 1 2 2 2 2 1 2 1 1 2 2 1 2 2 2

Within cluster sum of squares by cluster:
[1] 2888.221 3012.289
 (between_SS / total_SS =  22.9 %)

Available components:

[1] "cluster"      "centers"      "totss"        "withinss"     "tot.withinss"
[6] "betweenss"    "size"         "iter"         "ifault"

By performing K-means clustering with 2 centers, I result with two groups. The first cluster has no high average value for any of the themes with all of the average values ranging between 0-4. On the other hand, the second cluster has a high average value for the love theme, with an average value of 8.686. The first cluster has a size of 136 songs while the second cluster contains 83 songs.

set.seed(13)
kmeans(ts_dfm[,-1], centers = 3)

K-means clustering with 3 clusters of sizes 111, 70, 38

Cluster means:
      love     pain   change reflection
1 3.009009 2.729730 2.387387  0.9279279
2 9.057143 3.471429 3.228571  1.4285714
3 4.368421 8.894737 5.315789  1.3421053

Clustering vector:
  [1] 2 1 1 2 1 1 1 1 3 3 3 1 1 3 3 3 2 1 2 2 1 2 3 1 1 1 2 3 2 2 2 1 1 1 1 2 2
 [38] 1 1 1 1 1 1 3 2 2 2 2 1 1 1 1 3 2 1 1 2 2 2 2 1 1 3 1 1 3 1 2 3 2 2 3 2 1
 [75] 1 1 1 1 2 3 2 1 2 2 1 1 1 2 1 3 2 1 3 1 1 1 3 2 2 2 1 1 1 2 1 1 1 3 3 1 2
[112] 3 1 1 2 1 2 3 1 1 1 2 1 1 1 2 1 2 3 1 3 2 1 3 1 1 2 1 1 3 2 1 1 2 1 3 1 1
[149] 3 2 1 3 1 1 3 1 1 1 2 2 2 2 2 1 1 1 2 1 2 2 1 3 2 1 1 3 1 1 1 1 1 1 1 3 2
[186] 1 2 1 1 1 1 1 1 1 2 2 3 2 2 1 1 1 2 3 1 3 2 2 2 1 2 3 1 2 2 1 3 2 2

Within cluster sum of squares by cluster:
[1] 1238.649 1594.700 1859.184
 (between_SS / total_SS =  38.7 %)

Available components:

[1] "cluster"      "centers"      "totss"        "withinss"     "tot.withinss"
[6] "betweenss"    "size"         "iter"         "ifault"

By performing K-means clustering with 3 centers, I result with three clusters. The first cluster has no high average value for any of the cluster, consisting of 111 songs. The second group has a high average value for love, with an average value of 9.057. This cluster has 70 songs about it’s theme. The third cluster has a high average value of 8.894 for pain with 38 songs.

set.seed(13)
kmeans(ts_dfm[,-1], centers = 4)

K-means clustering with 4 clusters of sizes 10, 69, 30, 110

Cluster means:
      love     pain    change reflection
1 5.100000 4.600000 13.400000  1.6000000
2 8.971014 3.565217  2.971014  1.4202899
3 4.466667 9.766667  3.166667  1.2666667
4 3.000000 2.718182  2.354545  0.9272727

Clustering vector:
  [1] 2 4 4 1 4 4 4 4 3 3 3 4 4 3 3 3 2 4 2 2 4 2 3 4 4 4 2 3 2 2 2 4 4 4 4 2 2
 [38] 4 4 4 4 4 4 3 2 2 2 1 4 4 4 4 3 2 4 4 2 2 2 2 4 4 1 4 4 3 4 2 2 2 2 3 2 4
 [75] 4 4 4 4 2 1 2 4 2 2 4 4 4 2 4 3 2 4 3 4 4 4 1 2 2 2 4 4 4 2 4 4 4 3 4 4 2
[112] 2 4 4 2 4 2 3 4 4 4 2 4 4 4 2 4 2 3 4 1 1 4 3 4 4 2 4 4 3 2 4 4 2 4 3 4 4
[149] 3 2 4 3 4 4 1 4 4 4 2 2 2 2 2 4 4 3 2 4 2 2 4 3 2 4 4 3 4 1 4 4 4 4 4 3 2
[186] 4 2 4 4 4 4 4 4 4 2 2 3 2 2 4 4 4 2 3 4 1 2 2 2 4 2 3 4 2 2 4 3 2 2

Within cluster sum of squares by cluster:
[1]  530.1000 1385.6522  694.8667 1186.8545
 (between_SS / total_SS =  50.4 %)

Available components:

[1] "cluster"      "centers"      "totss"        "withinss"     "tot.withinss"
[6] "betweenss"    "size"         "iter"         "ifault"

With K-means clustering of 4 centers, I result with four groups. The first group has a high average value of 13.400 for the change category, consisting of 10 songs. The second cluster has a high average value of 8.971 for the love theme with 69 songs. The third cluster has a high average value of 9.766 for the pain theme with 30 songs. Lastly, the fourth cluster does not have any high average value for any of the four themes, consisting of 110 songs. With our 4 clusters, we develop a bracket for love songs, songs about pain/heartbreak, and songs about change. We then have one last group for songs that are undetected with any specific theme, a pattern similar to all of the other fittings we’ve done.

Using 4 centers seems to provide a better result than the previous two results as we now develop a category for another theme in our dictionary.

set.seed(13)
kmeans(ts_dfm[,-1], centers = 5)

K-means clustering with 5 clusters of sizes 7, 50, 26, 60, 76

Cluster means:
      love      pain    change reflection
1 5.714286  4.857143 15.428571  1.8571429
2 9.840000  3.960000  2.960000  1.5800000
3 4.653846 10.269231  2.884615  1.3846154
4 4.450000  1.500000  1.500000  1.1833333
5 2.815789  3.881579  3.578947  0.7236842

Clustering vector:
  [1] 2 5 4 1 5 4 5 4 3 3 3 5 5 3 3 3 2 4 2 2 5 2 3 5 4 5 2 3 2 2 2 5 4 5 5 4 4
 [38] 5 5 5 5 5 4 3 2 2 2 1 5 5 5 4 5 2 4 5 2 2 4 2 5 5 1 4 5 3 5 4 5 2 2 3 4 5
 [75] 4 5 5 5 4 5 2 5 5 2 4 5 4 2 4 3 4 5 3 5 5 4 5 4 4 4 4 5 4 2 4 4 5 3 5 4 2
[112] 5 4 4 2 4 2 3 5 4 4 2 4 5 4 2 4 2 3 5 1 1 5 3 5 5 2 4 4 3 2 4 5 4 4 5 4 5
[149] 5 2 5 3 4 4 1 4 4 4 2 5 2 5 2 5 5 3 2 4 2 2 4 3 4 5 4 5 5 5 4 5 4 4 5 3 2
[186] 4 2 5 5 5 5 5 5 5 2 2 3 2 2 5 4 5 2 3 4 1 2 2 2 5 2 3 5 2 4 5 3 5 2

Within cluster sum of squares by cluster:
[1]  418.8571 1012.7400  599.8077  613.8333  793.0789
 (between_SS / total_SS =  55.1 %)

Available components:

[1] "cluster"      "centers"      "totss"        "withinss"     "tot.withinss"
[6] "betweenss"    "size"         "iter"         "ifault"

With K-means clustering of 5 centers, I result with five groups. The first cluster has a high average value of 15.428 for the change theme, consisting of 7 songs. The second cluster of size 50 has a high average value for love with 9.840. The third cluster has a high average value for pain with 26 songs and an average of 10.269. The fourth cluster has a moderately high average value for love in comparison to the other themes, with an average value of 4.450, and containing 60 songs. Lastly, the fifth cluster has no high average value for any cluster and has a size of 76 songs left undetected of any specific theme. Rather than forming a category for our reflection theme, we instead develop another bracket for our love category, in which it is split between levels of strong and moderate love.

Examining the different number of centers we tuned, all models tend to result with a cluster that undetected a singular theme of a song. Pain, love, and change seem to be the most easily detected songs based on our models. Reflection is the one theme in our dictionary that has not been detected to its own cluster. Based on the results of our K-means modeling, I will proceed with k=4 centers since it provided the most interpretable results in relation to our project goal.

Running Multiple times

I will run K-means clustering with k=4 centers six times, to view the different clustering assignments I can achieve.

set.seed(13)
kmeans_ts1 <- kmeans(ts_dfm[,-1], centers = 4)
kmeans_ts2 <- kmeans(ts_dfm[,-1], centers = 4)
kmeans_ts3 <- kmeans(ts_dfm[,-1], centers = 4)
kmeans_ts4 <- kmeans(ts_dfm[,-1], centers = 4)
kmeans_ts5 <- kmeans(ts_dfm[,-1], centers = 4)
kmeans_ts6 <- kmeans(ts_dfm[,-1], centers = 4)

kmeans_ts1$centers

      love     pain    change reflection
1 5.100000 4.600000 13.400000  1.6000000
2 8.971014 3.565217  2.971014  1.4202899
3 4.466667 9.766667  3.166667  1.2666667
4 3.000000 2.718182  2.354545  0.9272727

kmeans_ts2$centers

      love     pain    change reflection
1 3.009174 2.724771  2.302752  0.9266055
2 9.075758 3.530303  2.772727  1.4696970
3 5.214286 4.357143 11.714286  1.2857143
4 4.466667 9.766667  3.166667  1.2666667

kmeans_ts3$centers

      love     pain    change reflection
1 3.009174 2.724771  2.302752  0.9266055
2 5.214286 4.357143 11.714286  1.2857143
3 4.466667 9.766667  3.166667  1.2666667
4 9.075758 3.530303  2.772727  1.4696970

kmeans_ts4$centers

      love     pain   change reflection
1 9.561404 3.438596 2.947368   1.035088
2 4.857143 2.857143 1.928571   6.000000
3 4.604651 8.186047 5.860465   1.116279
4 3.076190 2.819048 2.342857   0.600000

kmeans_ts5$centers

      love     pain    change reflection
1 5.214286 4.357143 11.714286  1.2857143
2 3.009174 2.724771  2.302752  0.9266055
3 4.466667 9.766667  3.166667  1.2666667
4 9.075758 3.530303  2.772727  1.4696970

kmeans_ts6$centers

       love     pain   change reflection
1 10.769231 3.846154 6.230769  1.3846154
2  7.258065 3.274194 2.000000  1.3709677
3  2.670103 2.659794 2.597938  0.8865979
4  4.264706 9.500000 4.558824  1.3823529

After running K-means clustering six times with 4 centers, I get that my 3/6 of my results produce the same results with same average values [kmeans_ts2, kmeans_ts3, kmeans_ts5]: one cluster with a high average for pain, one cluster with a high average for change, one cluster with a high average for love, and one cluster for all themes. kmeans_ts1 produces the same pattern of clusters, but just with different average values. On the other hand, kmeans_ts4 produces a cluster for love, pain, reflection, and one cluster for all remaining themes. This is new, since we have not seen any of our K-means clusters produce a cluster for reflection yet in our tuning process. kmeans_ts6 produces a different pattern of results: there exists a cluster with a high average value for love (strong love) and a cluster with a moderately high value for love (moderate love). Further, the other two clusters are for pain and a cluster for all themes unspecified.

Visualizing Results

I visualize these unique results with a scatterplot. With a focus on the count words for love and pain, since those have been revealed to be the most predominant features, I scale the count of love and pain dictionary words on the x- and y- axis. I then plot each song colored by what cluster group they fall into.

# plot for versions: kmeans_2, kmeans_3, kmeans_5
ggplot(cbind(ts_dfm, kmeans_ts2$cluster), aes(x = love, y = pain, color = factor(kmeans_ts2$cluster))) +
  geom_point(size = 0.9, shape = 6) + 
  scale_color_manual(limits = c(2, 4, 3, 1), values = c("pink", "navy", "purple", "gray"), name = "Theme", labels = c("Love", "Pain", "Change", "NA")) +
  theme_light() + 
  theme(axis.ticks.x = element_blank(), axis.ticks.y = element_blank(), plot.title = element_text(size=11.5, face = "bold"),     legend.position = c(1.01, 1.02),
    legend.justification = c("right", "top"),
    legend.box.just = "right",
    legend.margin = margin(6, 6, 6, 6),
    legend.text = element_text(size = 6),
    legend.box.background = element_rect(color="black", size=0.25)) +
  labs(title = "Frequency of Love vs Pain Words by Theme", 
       x = "Love", y = "Pain")

# plot for versions: kmeans_1
ggplot(cbind(ts_dfm, kmeans_ts1$cluster), aes(x = love, y = pain, color = factor(kmeans_ts1$cluster))) +
  geom_point(size = 0.9, shape = 6) + 
  scale_color_manual(limits = c(2, 3, 1, 4), values = c("pink", "navy", "purple", "gray"), name = "Theme", labels = c("Love", "Pain", "Change", "NA")) +
  theme_light() + 
  theme(axis.ticks.x = element_blank(), axis.ticks.y = element_blank(), plot.title = element_text(size=11.5, face = "bold"),     legend.position = c(1.01, 1.02),
    legend.justification = c("right", "top"),
    legend.box.just = "right",
    legend.margin = margin(6, 6, 6, 6),
    legend.text = element_text(size = 6),
    legend.box.background = element_rect(color="black", size=0.25)) +
  labs(title = "Frequency of Love vs Pain Words by Theme", 
       x = "Love", y = "Pain")

# plot for versions: kmeans_4
ggplot(cbind(ts_dfm, kmeans_ts4$cluster), aes(x = love, y = pain, color = factor(kmeans_ts4$cluster))) +
  geom_point(size = 0.9, shape = 6) + 
  scale_color_manual(limits = c(1, 3, 2, 4), values = c("pink", "navy", "gold", "gray"), name = "Theme", labels = c("Love", "Pain", "Reflection", "NA")) +
  theme_light() + 
  theme(axis.ticks.x = element_blank(), axis.ticks.y = element_blank(), plot.title = element_text(size=11.5, face = "bold"),     legend.position = c(1.01, 1.02),
    legend.justification = c("right", "top"),
    legend.box.just = "right",
    legend.margin = margin(6, 6, 6, 6),
    legend.text = element_text(size = 6),
    legend.box.background = element_rect(color="black", size=0.25)) +
  labs(title = "Frequency of Love vs Pain Words by Theme", 
       x = "Love", y = "Pain")

# plot for version: kmeans_6
ggplot(cbind(ts_dfm, kmeans_ts6$cluster), aes(x = love, y = pain, color = factor(kmeans_ts6$cluster))) +
  geom_point(size = 0.9, shape = 6) + 
  scale_color_manual(limits = c(1, 2, 4, 3), values = c("pink", "maroon", "navy", "gray"), name = "Theme", labels = c("Strong Love", "Moderate Love", "Pain", "NA")) +
  theme_light() + 
  theme(axis.ticks.x = element_blank(), axis.ticks.y = element_blank(), plot.title = element_text(size=11.5, face = "bold"),     legend.position = c(1.01, 1.02),
    legend.justification = c("right", "top"),
    legend.box.just = "right",
    legend.margin = margin(6, 6, 6, 6),
    legend.text = element_text(size = 6), 
    legend.box.background = element_rect(color="black", size=0.25)) +
  labs(title = "Frequency of Love vs Pain Words by Theme", 
       x = "Love", y = "Pain")

Interpreting the change amongst the plots, the most detectable change between the first two plots is that one single love song becomes a change song, implying how the second iteration of K-means clustering collected more change songs. The shift to the third plot shows the development of undetected songs into reflection songs, as well as more pain songs being identified from the undetected population. Lastly, the fourth plot shows how the centered love songs are now classified as moderate love, as their count for love words are not as high in comparison to the other songs in their theme.

Hierarchical Clustering

Let’s try something else. Hierarchical clustering is an unsupervised method for clustering observations. It is an alternative approach to K-Means clustering because it does not require a pre-specified number of clusters. There are two types of hierarchical clustering algorithms: agglomerative clustering and divisive hierarchical clustering.

Agglomerative clustering is more appropriate for this project because it starts with individual data points as clusters and progressively merge them, which can be computationally less intensive and more intuitive. It is also flexible with the number of clusters, which we apply as the 4 themes in this case.

Agglomerative Nesting (AGNES)

This algorithm works “bottom-up.” Each observation is initially considered as a single-element cluster; at each step, the two clusters that are the most similar are combined into a new bigger cluster. This procedure is repeated until all observations are members of just one single root cluster.

As we perform hierarchical clustering, we observe the dissimilarity measure between each pair of observations. As we repeat the procedure, we extend this dissimilarity to a pair of groups of observations, i.e. linkage. In my project, I consider the following types of linkage: complete (maximal intercluster dissimilarity), single (minimal intercluster dissimilarity), and average (mean intercluster dissimilarity).

Visualizing Dendrograms

I perform hierarchical cluster analysis with the hclust() function:

# create different hierarchical clusterings
hcluster_complete_ts <- hclust(dist(ts_dfm), method = "complete")
hcluster_single_ts <- hclust(dist(ts_dfm), method = "single")
hcluster_average_ts <- hclust(dist(ts_dfm), method = "average")

plot(hcluster_complete_ts, main = "Complete Linkage",
     xlab = "", sub = "", cex = 0.17)

plot(hcluster_single_ts, main = "Single Linkage",
     xlab = "", sub = "", cex = .17)

plot(hcluster_average_ts, main = "Average Linkage",
     xlab = "", sub = "", cex = .17)

Choosing Linkage type

The cutree() function determines the cluster label for each observation by a given cut of the dendrogram (in this case I am cutting 4 clusters for our four themes).

cutree(hcluster_complete_ts, 4)

  [1] 1 2 2 3 2 2 2 2 4 4 4 2 2 4 4 4 1 2 1 1 2 1 2 4 2 2 2 4 2 2 1 2 2 2 2 2 2
 [38] 2 4 4 2 2 2 4 1 2 1 3 2 2 2 2 2 1 2 2 1 1 2 2 2 2 3 2 2 2 2 2 2 2 2 4 2 2
 [75] 2 2 2 2 2 3 1 2 2 2 2 2 2 2 2 4 2 2 4 2 2 2 3 2 2 2 2 2 2 2 2 2 2 4 2 2 1
[112] 2 2 2 1 2 2 4 2 2 2 2 2 2 2 2 2 1 4 2 3 3 2 4 2 2 1 2 2 4 2 2 2 2 2 4 2 4
[149] 2 2 2 4 2 2 3 2 2 2 2 2 1 2 2 4 2 4 2 2 2 1 2 4 2 2 2 4 4 3 2 2 2 2 2 2 1
[186] 2 1 4 2 2 2 2 2 2 2 2 4 2 1 2 2 2 2 4 2 3 1 2 1 2 1 4 2 1 2 2 4 2 3

cutree(hcluster_single_ts, 4)

  [1] 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
 [38] 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 1 1 1 1 1 1 1 1 1 1 1
 [75] 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
[112] 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
[149] 1 1 1 1 1 1 3 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
[186] 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 4 1 1 1 1 1 1 1 1 1 1 1 1 1

cutree(hcluster_average_ts, 4)

  [1] 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
 [38] 1 1 1 1 1 1 1 1 1 1 2 1 1 1 1 1 1 1 1 1 1 1 1 1 1 3 1 1 1 1 1 1 1 1 1 1 1
 [75] 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 4 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
[112] 1 1 1 1 1 1 4 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 4 1 1 1 1 1 1 1 1
[149] 1 1 1 1 1 1 3 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
[186] 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 3 1 1 1 1 1 1 1 1 1 1 1 1 1

I find that the complete linkage hierarchical clustering provides me with the most diverse clusters, so I will utilize this type of linkage moving forward. The single and average clusters seem to cluster all songs into the same cluster (1), which could be reflective of how we saw that the majority of songs were placed in the undefined theme category in the K-means clustering.

Tuning Number of Clusters

Now that I’ve chosen which linkage type to use, I proceed with tuning various number of clusters to see how I could best cut the tree to model my data.

4 Clusters?

To examine what the most accurate number of cuts to make, I have to look at what songs exactly are being stored in 4 clusters. Understanding the songs being put into these clusters can tell me what themes each of them are implicitly representing.

table(cutree(hcluster_complete_ts, 4))


  1   2   3   4 
 25 151  11  32

At a glance, I observe that cluster 1 has 25 songs, cluster 2 has the bulk of 151 songs, cluster 3 has 11 songs, and cluster 4 has 32 songs.

knitr::kable(official_list[which(cutree(hcluster_complete_ts, 4) == 1), 1]) %>% 
  kable_styling(full_width = F) %>% 
  scroll_box(width = "30%", height = "200px")

x
…Ready For It?
Babe (Taylor’s Version) (From The Vault)
Bad Blood (feat. Kendrick Lamar) (Taylor’s Version)
Bad Blood (Taylor’s Version)
Bejeweled
cardigan
cowboy like me
Cruel Summer - Live from TS \| The Eras Tour
Don’t Blame Me
Dress
Electric Touch (feat. Fall Out Boy) (Taylor’s Version) (From The Vault)
Hits Different
London Boy
Love Story (Taylor’s Version)
Miss Americana & The Heartbreak Prince
Only The Young - Featured in Miss Americana
Sparks Fly (Taylor’s Version)
Superstar (Taylor’s Version)
The Lucky One (Taylor’s Version)
The Moment I Knew (Taylor’s Version)
Today Was A Fairytale (Taylor’s Version)
When Emma Falls in Love (Taylor’s Version) (From The Vault)
Wildest Dreams (Taylor’s Version)
Wonderland (Taylor’s Version)
You Are In Love (Taylor’s Version)

knitr::kable(official_list[which(cutree(hcluster_complete_ts, 4) == 2), 1]) %>% 
  kable_styling(full_width = F) %>% 
  scroll_box(width = "30%", height = "200px")

x
’tis the damn season
“Slut!” (Taylor’s Version) (From The Vault)
A Perfectly Good Heart
A Place in this World
Afterglow
All Of The Girls You Loved Before
All Too Well (Taylor’s Version)
All You Had To Do Was Stay (Taylor’s Version)
Back To December (Taylor’s Version)
Begin Again (Taylor’s Version)
Better Man (Taylor’s Version) (From The Vault)
betty
Bigger Than The Whole Sky
Blank Space (Taylor’s Version)
Bye Bye Baby (Taylor’s Version) (From The Vault)
Call It What You Want
Carolina - From The Motion Picture “Where The Crawdads Sing”
Castles Crumbling (feat. Hayley Williams) (Taylor’s Version) (From The Vault)
champagne problems
Change (Taylor’s Version)
Christmas Tree Farm
Christmas Tree Farm (Old Timey Version)
Clean (Taylor’s Version)
Come Back…Be Here (Taylor’s Version)
Come In With The Rain (Taylor’s Version)
coney island (feat. The National)
Cruel Summer
Daylight
Dear John (Taylor’s Version)
Dear Reader
Death By A Thousand Cuts
Delicate
Don’t You (Taylor’s Version) (From The Vault)
dorothea
Enchanted (Taylor’s Version)
End Game
epiphany
evermore (feat. Bon Iver)
exile (feat. Bon Iver)
Eyes Open (Taylor’s Version)
False God
Fearless (Taylor’s Version)
Fifteen (Taylor’s Version)
Foolish One (Taylor’s Version) (From The Vault)
Forever & Always (Taylor’s Version)
Forever Winter (Taylor’s Version) (From The Vault)
Girl At Home (Taylor’s Version)
Glitch
gold rush
Gorgeous
happiness
Haunted (Taylor’s Version)
Hey Stephen (Taylor’s Version)
hoax
Holy Ground (Taylor’s Version)
How You Get The Girl (Taylor’s Version)
I Almost Do (Taylor’s Version)
I Bet You Think About Me (feat. Chris Stapleton) (Taylor’s Version) (From The Vault)
I Can See You (Taylor’s Version) (From The Vault)
I Did Something Bad
I Forgot That You Existed
I Know Places (Taylor’s Version)
I Think He Knows
I’m Only Me When I’m With You
If This Was A Movie (Taylor’s Version)
illicit affairs
Invisible
invisible string
Is It Over Now? (Taylor’s Version) (From The Vault)
It’s Nice To Have A Friend
it’s time to go - bonus track
ivy
Jump Then Fall (Taylor’s Version)
Karma
Karma (feat. Ice Spice)
King Of My Heart
Last Kiss (Taylor’s Version)
Lavender Haze
Long Live (Taylor’s Version)
long story short
Look What You Made Me Do
Lover
mad woman
Maroon
Mary’s Song (Oh My My My)
Mastermind
ME! (feat. Brendon Urie of Panic! At The Disco)
Mean (Taylor’s Version)
Message In A Bottle (Taylor’s Version) (From The Vault)
Midnight Rain
Mine (Taylor’s Version)
mirrorball
my tears ricochet
New Year’s Day
Nothing New (feat. Phoebe Bridgers) (Taylor’s Version) (From The Vault)
Now That We Don’t Talk (Taylor’s Version) (From The Vault)
Our Song
Ours (Taylor’s Version)
Paper Rings
Paris
peace
Picture To Burn
Question…?
Renegade - Pop Version
Ronan (Taylor’s Version)
Run (feat. Ed Sheeran) (Taylor’s Version) (From The Vault)
Sad Beautiful Tragic (Taylor’s Version)
Say Don’t Go (Taylor’s Version) (From The Vault)
seven
Should’ve Said No
Snow On The Beach (feat. Lana Del Rey)
Snow On The Beach (feat. More Lana Del Rey)
So It Goes…
Soon You’ll Get Better (feat. The Chicks)
Speak Now (Taylor’s Version)
Starlight (Taylor’s Version)
Stay Beautiful
Style (Taylor’s Version)
Suburban Legends (Taylor’s Version) (From The Vault)
Superman (Taylor’s Version)
Sweet Nothing
Teardrops on My Guitar - Pop Version
Tell Me Why (Taylor’s Version)
That’s When (feat. Keith Urban) (Taylor’s Version) (From The Vault)
The Great War
The Joker And The Queen (feat. Taylor Swift)
the lakes - bonus track
the lakes - original version
the last great american dynasty
The Last Time (feat. Gary Lightbody of Snow Patrol) (Taylor’s Version)
The Man
The Outside
The Story Of Us (Taylor’s Version)
The Very First Night (Taylor’s Version) (From The Vault)
The Way I Loved You (Taylor’s Version)
this is me trying
This Is Why We Can’t Have Nice Things
This Love (Taylor’s Version)
Tied Together with a Smile
Timeless (Taylor’s Version) (From The Vault)
tolerate it
Treacherous (Taylor’s Version)
Untouchable (Taylor’s Version)
Vigilante Shit
We Were Happy (Taylor’s Version) (From The Vault)
White Horse (Taylor’s Version)
willow
You All Over Me (feat. Maren Morris) (Taylor’s Version) (From The Vault)
You Belong With Me (Taylor’s Version)
You Need To Calm Down
You’re Not Sorry (Taylor’s Version)

knitr::kable(official_list[which(cutree(hcluster_complete_ts, 4) == 3), 1]) %>% 
  kable_styling(full_width = F) %>% 
  scroll_box(width = "30%", height = "200px")

x
22 (Taylor’s Version)
Dancing With Our Hands Tied
Everything Has Changed (feat. Ed Sheeran) (Taylor’s Version)
High Infidelity
Innocent (Taylor’s Version)
Never Grow Up (Taylor’s Version)
New Romantics (Taylor’s Version)
Shake It Off (Taylor’s Version)
The Best Day (Taylor’s Version)
Welcome To New York (Taylor’s Version)
You’re On Your Own, Kid

knitr::kable(official_list[which(cutree(hcluster_complete_ts, 4) == 4), 1]) %>% 
  kable_styling(full_width = F) %>% 
  scroll_box(width = "30%", height = "200px")

x
All Too Well (10 Minute Version) (Taylor’s Version) (From The Vault)
All Too Well (10 Minute Version) (The Short Film)
All Too Well (Sad Girl Autumn Version) - Recorded at Long Pond Studios
Anti-Hero
Anti-Hero (feat. Bleachers)
august
Better Than Revenge (Taylor’s Version)
Breathe (feat. Colbie Caillat) (Taylor’s Version)
closure
Cold As You
Cornelia Street
Getaway Car
I Knew You Were Trouble (Taylor’s Version)
I Wish You Would (Taylor’s Version)
Labyrinth
marjorie
Mr. Perfectly Fine (Taylor’s Version) (From The Vault)
no body, no crime (feat. HAIM)
Out Of The Woods (Taylor’s Version)
Red (Taylor’s Version)
right where you left me - bonus track
Safe & Sound (feat. Joy Williams and John Paul White) (Taylor’s Version)
State Of Grace (Taylor’s Version)
Stay Stay Stay (Taylor’s Version)
Sweeter Than Fiction - From “One Chance” Soundtrack
the 1
The Archer
The Other Side Of The Door (Taylor’s Version)
Tim McGraw
We Are Never Ever Getting Back Together (Taylor’s Version)
Would’ve, Could’ve, Should’ve
You’re Losing Me (From The Vault)

The first cluster contains dominantly love songs, evidently by When Emma Falls in Love (Taylor’s Version), Your Are In Love (Taylor’s Version), and Love Story (Taylor’s Version). However, it does contain some songs that are not about love, but may have been included based on the predefined terms in our dictionary, such as Babe or Bad Blood.

The second cluster contains a majority of songs, consisting of both love songs and heartbreak songs (You Belong With Me (Taylor’s Version), Vigilante Shit, etc.). I deduce this cluster is the unspecified cluster of any one particular theme.

The third cluster contains an accurate depiction of the change theme: Everything Has Changed (Taylor’s Version), Never Grow Up (Taylor’s Version), The Best Day (Taylor’s Version).

The fourth cluster contains mostly songs about pain, with the exception of some songs (which could be due to how the dictionary was formulated): All Too Well (10 Minute Version) (Taylor’s Version), Better Than Revenge (Taylor’s Version), Would’ve, Could’ve, Should’ve, You’re Losing Me.

5 Clusters?

I will now try cutting the tree at 5 clusters to see how this could change our results.

cutree(hcluster_complete_ts, 5)

  [1] 1 2 2 3 2 2 2 2 4 4 4 2 2 4 4 4 1 2 1 1 2 1 2 4 2 2 2 4 2 2 1 2 2 2 2 2 2
 [38] 2 4 4 2 2 2 4 1 2 1 3 2 2 2 2 2 1 2 2 1 1 2 2 2 2 5 2 2 2 2 2 2 2 2 4 2 2
 [75] 2 2 2 2 2 3 1 2 2 2 2 2 2 2 2 4 2 2 4 2 2 2 3 2 2 2 2 2 2 2 2 2 2 4 2 2 1
[112] 2 2 2 1 2 2 4 2 2 2 2 2 2 2 2 2 1 4 2 3 3 2 4 2 2 1 2 2 4 2 2 2 2 2 4 2 4
[149] 2 2 2 4 2 2 5 2 2 2 2 2 1 2 2 4 2 4 2 2 2 1 2 4 2 2 2 4 4 3 2 2 2 2 2 2 1
[186] 2 1 4 2 2 2 2 2 2 2 2 4 2 1 2 2 2 2 4 2 5 1 2 1 2 1 4 2 1 2 2 4 2 3

table(cutree(hcluster_complete_ts, 5))


  1   2   3   4   5 
 25 151   8  32   3

knitr::kable(official_list[which(cutree(hcluster_complete_ts, 5) == 1), 1]) %>% 
  kable_styling(full_width = F) %>% 
  scroll_box(width = "30%", height = "200px")

x
…Ready For It?
Babe (Taylor’s Version) (From The Vault)
Bad Blood (feat. Kendrick Lamar) (Taylor’s Version)
Bad Blood (Taylor’s Version)
Bejeweled
cardigan
cowboy like me
Cruel Summer - Live from TS \| The Eras Tour
Don’t Blame Me
Dress
Electric Touch (feat. Fall Out Boy) (Taylor’s Version) (From The Vault)
Hits Different
London Boy
Love Story (Taylor’s Version)
Miss Americana & The Heartbreak Prince
Only The Young - Featured in Miss Americana
Sparks Fly (Taylor’s Version)
Superstar (Taylor’s Version)
The Lucky One (Taylor’s Version)
The Moment I Knew (Taylor’s Version)
Today Was A Fairytale (Taylor’s Version)
When Emma Falls in Love (Taylor’s Version) (From The Vault)
Wildest Dreams (Taylor’s Version)
Wonderland (Taylor’s Version)
You Are In Love (Taylor’s Version)

knitr::kable(official_list[which(cutree(hcluster_complete_ts, 5) == 2), 1]) %>% 
  kable_styling(full_width = F) %>% 
  scroll_box(width = "30%", height = "200px")

x
’tis the damn season
“Slut!” (Taylor’s Version) (From The Vault)
A Perfectly Good Heart
A Place in this World
Afterglow
All Of The Girls You Loved Before
All Too Well (Taylor’s Version)
All You Had To Do Was Stay (Taylor’s Version)
Back To December (Taylor’s Version)
Begin Again (Taylor’s Version)
Better Man (Taylor’s Version) (From The Vault)
betty
Bigger Than The Whole Sky
Blank Space (Taylor’s Version)
Bye Bye Baby (Taylor’s Version) (From The Vault)
Call It What You Want
Carolina - From The Motion Picture “Where The Crawdads Sing”
Castles Crumbling (feat. Hayley Williams) (Taylor’s Version) (From The Vault)
champagne problems
Change (Taylor’s Version)
Christmas Tree Farm
Christmas Tree Farm (Old Timey Version)
Clean (Taylor’s Version)
Come Back…Be Here (Taylor’s Version)
Come In With The Rain (Taylor’s Version)
coney island (feat. The National)
Cruel Summer
Daylight
Dear John (Taylor’s Version)
Dear Reader
Death By A Thousand Cuts
Delicate
Don’t You (Taylor’s Version) (From The Vault)
dorothea
Enchanted (Taylor’s Version)
End Game
epiphany
evermore (feat. Bon Iver)
exile (feat. Bon Iver)
Eyes Open (Taylor’s Version)
False God
Fearless (Taylor’s Version)
Fifteen (Taylor’s Version)
Foolish One (Taylor’s Version) (From The Vault)
Forever & Always (Taylor’s Version)
Forever Winter (Taylor’s Version) (From The Vault)
Girl At Home (Taylor’s Version)
Glitch
gold rush
Gorgeous
happiness
Haunted (Taylor’s Version)
Hey Stephen (Taylor’s Version)
hoax
Holy Ground (Taylor’s Version)
How You Get The Girl (Taylor’s Version)
I Almost Do (Taylor’s Version)
I Bet You Think About Me (feat. Chris Stapleton) (Taylor’s Version) (From The Vault)
I Can See You (Taylor’s Version) (From The Vault)
I Did Something Bad
I Forgot That You Existed
I Know Places (Taylor’s Version)
I Think He Knows
I’m Only Me When I’m With You
If This Was A Movie (Taylor’s Version)
illicit affairs
Invisible
invisible string
Is It Over Now? (Taylor’s Version) (From The Vault)
It’s Nice To Have A Friend
it’s time to go - bonus track
ivy
Jump Then Fall (Taylor’s Version)
Karma
Karma (feat. Ice Spice)
King Of My Heart
Last Kiss (Taylor’s Version)
Lavender Haze
Long Live (Taylor’s Version)
long story short
Look What You Made Me Do
Lover
mad woman
Maroon
Mary’s Song (Oh My My My)
Mastermind
ME! (feat. Brendon Urie of Panic! At The Disco)
Mean (Taylor’s Version)
Message In A Bottle (Taylor’s Version) (From The Vault)
Midnight Rain
Mine (Taylor’s Version)
mirrorball
my tears ricochet
New Year’s Day
Nothing New (feat. Phoebe Bridgers) (Taylor’s Version) (From The Vault)
Now That We Don’t Talk (Taylor’s Version) (From The Vault)
Our Song
Ours (Taylor’s Version)
Paper Rings
Paris
peace
Picture To Burn
Question…?
Renegade - Pop Version
Ronan (Taylor’s Version)
Run (feat. Ed Sheeran) (Taylor’s Version) (From The Vault)
Sad Beautiful Tragic (Taylor’s Version)
Say Don’t Go (Taylor’s Version) (From The Vault)
seven
Should’ve Said No
Snow On The Beach (feat. Lana Del Rey)
Snow On The Beach (feat. More Lana Del Rey)
So It Goes…
Soon You’ll Get Better (feat. The Chicks)
Speak Now (Taylor’s Version)
Starlight (Taylor’s Version)
Stay Beautiful
Style (Taylor’s Version)
Suburban Legends (Taylor’s Version) (From The Vault)
Superman (Taylor’s Version)
Sweet Nothing
Teardrops on My Guitar - Pop Version
Tell Me Why (Taylor’s Version)
That’s When (feat. Keith Urban) (Taylor’s Version) (From The Vault)
The Great War
The Joker And The Queen (feat. Taylor Swift)
the lakes - bonus track
the lakes - original version
the last great american dynasty
The Last Time (feat. Gary Lightbody of Snow Patrol) (Taylor’s Version)
The Man
The Outside
The Story Of Us (Taylor’s Version)
The Very First Night (Taylor’s Version) (From The Vault)
The Way I Loved You (Taylor’s Version)
this is me trying
This Is Why We Can’t Have Nice Things
This Love (Taylor’s Version)
Tied Together with a Smile
Timeless (Taylor’s Version) (From The Vault)
tolerate it
Treacherous (Taylor’s Version)
Untouchable (Taylor’s Version)
Vigilante Shit
We Were Happy (Taylor’s Version) (From The Vault)
White Horse (Taylor’s Version)
willow
You All Over Me (feat. Maren Morris) (Taylor’s Version) (From The Vault)
You Belong With Me (Taylor’s Version)
You Need To Calm Down
You’re Not Sorry (Taylor’s Version)

knitr::kable(official_list[which(cutree(hcluster_complete_ts, 5) == 3), 1]) %>% 
  kable_styling(full_width = F) %>% 
  scroll_box(width = "30%", height = "200px")

x
22 (Taylor’s Version)
Dancing With Our Hands Tied
High Infidelity
Innocent (Taylor’s Version)
Never Grow Up (Taylor’s Version)
New Romantics (Taylor’s Version)
The Best Day (Taylor’s Version)
You’re On Your Own, Kid

knitr::kable(official_list[which(cutree(hcluster_complete_ts, 5) == 4), 1]) %>% 
  kable_styling(full_width = F) %>% 
  scroll_box(width = "30%", height = "200px")

x
All Too Well (10 Minute Version) (Taylor’s Version) (From The Vault)
All Too Well (10 Minute Version) (The Short Film)
All Too Well (Sad Girl Autumn Version) - Recorded at Long Pond Studios
Anti-Hero
Anti-Hero (feat. Bleachers)
august
Better Than Revenge (Taylor’s Version)
Breathe (feat. Colbie Caillat) (Taylor’s Version)
closure
Cold As You
Cornelia Street
Getaway Car
I Knew You Were Trouble (Taylor’s Version)
I Wish You Would (Taylor’s Version)
Labyrinth
marjorie
Mr. Perfectly Fine (Taylor’s Version) (From The Vault)
no body, no crime (feat. HAIM)
Out Of The Woods (Taylor’s Version)
Red (Taylor’s Version)
right where you left me - bonus track
Safe & Sound (feat. Joy Williams and John Paul White) (Taylor’s Version)
State Of Grace (Taylor’s Version)
Stay Stay Stay (Taylor’s Version)
Sweeter Than Fiction - From “One Chance” Soundtrack
the 1
The Archer
The Other Side Of The Door (Taylor’s Version)
Tim McGraw
We Are Never Ever Getting Back Together (Taylor’s Version)
Would’ve, Could’ve, Should’ve
You’re Losing Me (From The Vault)

knitr::kable(official_list[which(cutree(hcluster_complete_ts, 5) == 5), 1]) %>% 
  kable_styling(full_width = F) %>% 
  scroll_box(width = "30%", height = "200px")

x
Everything Has Changed (feat. Ed Sheeran) (Taylor’s Version)
Shake It Off (Taylor’s Version)
Welcome To New York (Taylor’s Version)

Again, by analyzing the clusters based on what songs are in each category, I can determine what themes are being implicitly assigned.

The first cluster contains songs about love featuring: Wildest Dreams (Taylor’s Version) and You Are In Love (Taylor’s Version).
The second cluster contains the majority of songs, which are undetected of one particular theme.
The third cluster features: Never Grow Up (Taylor’s Version), The Best Day (Taylor’s Version), and 22 (Taylor’s Version).
The fourth cluster contains some song on pain, featuring: You’re Losing Me, Cold As You, and I Knew You Were Trouble (Taylor’s Version).
The fifth cluster features: Everything Has Changed (Taylor’s Version), Shake It Off (Taylor’s Version), and Welcome To New York (Taylor’s Version).

Examining the third and fifth cluster, I see a mix of songs about change and reflection in both. Although I initially planned the reflection category to fall more in lines with her reputation and response to self-image/media, I can understand the interpretation of You’re On Your Own, Kid as either a change song or reflection song. I will identify the third cluster a change and the fourth cluster as reflection; primarily because Shake It Off (Taylor’s Version) is about letting go of negativity and criticism.

By examining the table of both results, I notice that changing the number of clusters seems to split the 11 songs in the change theme into two new clusters: one with a size of 8 and the other with size 3. I identified above that one of these split clusters could be interpreted as a reflection theme.

6 Clusters?

Now, lets try this one more time! Let’s see what would happen if we cut our tree into 6 clusters.

cutree(hcluster_complete_ts, 6)

  [1] 1 2 2 3 2 2 2 2 4 4 4 2 2 4 4 4 1 2 1 1 2 1 2 4 2 2 2 4 2 2 1 2 2 2 2 2 2
 [38] 2 4 4 2 2 2 4 1 2 1 3 2 2 2 2 2 1 2 2 1 1 2 2 2 2 5 2 2 2 2 2 2 2 2 4 2 2
 [75] 2 2 2 2 2 3 1 2 2 2 2 2 2 2 2 4 2 2 6 2 2 2 3 2 2 2 2 2 2 2 2 2 2 4 2 2 1
[112] 2 2 2 1 2 2 6 2 2 2 2 2 2 2 2 2 1 4 2 3 3 2 4 2 2 1 2 2 6 2 2 2 2 2 4 2 4
[149] 2 2 2 4 2 2 5 2 2 2 2 2 1 2 2 4 2 4 2 2 2 1 2 4 2 2 2 4 4 3 2 2 2 2 2 2 1
[186] 2 1 4 2 2 2 2 2 2 2 2 4 2 1 2 2 2 2 4 2 5 1 2 1 2 1 4 2 1 2 2 4 2 3

table(cutree(hcluster_complete_ts, 6))


  1   2   3   4   5   6 
 25 151   8  29   3   3

knitr::kable(official_list[which(cutree(hcluster_complete_ts, 6) == 6), 1]) %>% 
  kable_styling(full_width = F) %>% 
  scroll_box(width = "30%", height = "200px")

x
I Wish You Would (Taylor’s Version)
marjorie
Out Of The Woods (Taylor’s Version)

The table shows that the clusters from before are remained, but the pain cluster is removed of songs: I Wish You Would (Taylor’s Version), marjorie, and Out Of The Woods (Taylor’s Version).

Since these songs should still be identified in the pain category, I will select the tuned tree at 5 clusters, since there will be a cluster for each theme along with the additional undetected bucket.

hclust5_ts <- cutree(hcluster_complete_ts, 5)

Model Accuracies

To compare my models: the K- Means clustering with 4 centers and Hierarchical Agglomerative clustering with 5 clusters, I use the plotly package to examine the songs contained in each of the theme per model. I visualize the songs per theme using a bar chart, with a Plotly hover feature that shows the songs placed in each theme.

Results of the Best Model

One distinct observation is that the Hierarchical clustering is able to interpret a small category for the reflection theme, one in which the K-means clustering ignores. On the other hand, the number of songs in the undistingushed theme is much greater in the Hierarchical clustering than in the K-means clustering- this is due to more songs being identified as love songs in K-means. Another distinct observation is that out of specified themes, the K-means has the greatest size of songs in the love category, while the Hierarchical clustering was able to label more songs into pain than love.

Some observations between the placement of songs between models:

Call It What You Want, Cruel Summer, Enchanted, End Game, Hey Stephen, Style, This Love is accurately identified as a love song in K-means, but undetected as any theme in hierarchical.
Delicate is inaccurately identified as a song about pain in K-means but rather unspecified in hierarchical (the song should be placed in the love theme).
Ronan (Taylor’s Version) and The Last Time (Taylor’s Version) were accurately defined as a song about pain in K-means, but unspecified in hierarchical.
Labyrinth was inaccurately categorized as a song about pain in both algorithms, when it should be classified as a love song.
The majority of songs identified in the love, pain, and change themes in hierarchical are also labelled same in K-means [i.e. the placement of Would’ve, Could’ve, Should’ve, You’re Losing Me, Wildest Dreams]

Conclusion

Overall, the K-means clustering is a better fit to the data because it detects the theme of more songs. I would justify that both models have a similar error rate when it comes to predicting the theme of the song, but the K-means would determine more songs as love or pain rather than keeping it as undetected. The K-means had a size of 109 undetected songs, while Hierarchical had a size of 151 undetected songs. Although Hierarchical was able to produce a somewhat interpretable bucket for reflection, it did not satisfy the original definition for what was to be deemed as reflection/reputation/response-to-media.

Challenges

While we were able to create a model that predicts the theme of Taylor Swift’s discography, there were some setbacks that we have to consider.

Our algorithms work on the premise that documents with the same topic have the same dictionary of words and that they are bag-of-word models, meaning that they only consider individual tokens of words, rather than the poetry of the lyrics and how the words are used together. Literary devices that Swift uses such as paradox, humor, or irony will not be captured in our model.
Our dictionary of words is limited. While I was curating my dictionary of words, I extensively looked at the words in songs from various themes so that I could best capture how Swift uses her language for different discussions. I referenced specific songs that I thought well-highlighted a theme: for instance, Welcome to New York (Taylor’s Version) I would say was overfitted into our dictionary for change, based on the specific word of “new” and “york”. In addition, the use of words in different contexts is inevitable- the word “cruel” is used in a love sense in Cruel Summer but utilized in a pain sense in All Too Well (Taylor’s Version) or Mr. Perfectly Fine (Taylor’s Version). I chose which dictionary the word would fall into based on which theme of song it fell more into.
We are not looking at the entire lyrics. Since we are using Musixmatch’s API data, we only had access to the first 30% of the lyrics. As a songwriter, Swift conveys a story in her songs which would explain why the beginning of her song may not necessarily capture the emotional context of the entire track. With the first few lyrics intended towards setting up the narrative or background, it will not capture the entirety of the song’s theme.

Wrapping Up

Nonetheless, our K-means cluster was able to detect songs that were strongly about love, pain, and change. This was not a delicate task to ask for our machine to solve: detecting the theme of a Taylor Swift song based on only the first 30% of the song’s lyrical data. Although it didn’t reach the endgame of accurately labeling the themes for all songs, it still accomplished for a select number of songs, all too well. Moving forward, we could look into sentiment analysis in order to study the mood of the songs, which could lead into a more intricate understanding of the theme. Furthermore, incorporating the acousticness or danceability (as provided in Spotify API) could also be a useful feature in determining the mood and possibly theme. All in all, I would say that our project was a success: we had fun and we learned a lot unsupervised!

Is It Over Now?

Next time you listen to Taylor Swift, take the moment to indulge the poetry and creativity she so eloquently evokes in her mastery, refined songwriting.

The Tortured Poets Department releases April 19!

Code Appendix

knitr::opts_chunk$set(message = FALSE,
                      warning = FALSE,
                      comment = NA)

library(tidyverse)
library(tidymodels)
library(dplyr)
library(ggplot2)
library(plotly)
library(naniar)
library(quanteda)
library(quanteda.textstats)
library(wordcloud)
library(kableExtra)
# spotify packages
library(ggridges)
library(spotifyr)
# tidying data packages
library(stopwords) # stopwords
library(SnowballC) # word stems
library(tm)
library(tokenizers)
# clustering
library(tidyclust)
library(GGally)

Sys.setenv(SPOTIFY_CLIENT_ID = '90eebf23819f4f6a9a3c6b26f8ebedd2')
Sys.setenv(SPOTIFY_CLIENT_SECRET = 'b17dc3bbb1744a0f9bb0a5725cbcae07')

# spotify access token
access_token <- get_spotify_access_token()

# musixmatch access token
api_key <- '86c81cb48352f6946ef78b43cbe61ad9'

# set seed for dataset previews 
set.seed(13)
knitr::include_graphics(path = 'taylorswift1.jpg')
knitr::include_graphics(path = 'spotifylogo.png')
knitr::include_graphics(path = 'tscovers.jpeg')
ts_albums <- get_artist_audio_features(artist = 'taylor swift',
                                include_groups = "album")

ts_singles <- get_artist_audio_features(artist = 'taylor swift',
                                include_groups = "single")

ts_discography <- full_join(ts_albums %>%
  select(track_name, track_number, album_name, album_type, album_release_date, 
         danceability, energy, key, loudness, mode, speechiness, 
         acousticness, instrumentalness, liveness, valence, tempo,
         time_signature, key_mode, duration_ms, explicit), 
  ts_singles %>%
  select(track_name, track_number, album_name, album_type, album_release_date, 
         danceability, energy, key, loudness, mode, speechiness, 
         acousticness, instrumentalness, liveness, valence, tempo,
         time_signature, key_mode, duration_ms, explicit)) %>%
  arrange(desc(album_release_date)) 
ts <- ts_discography %>%
  filter(album_release_date > '2017-11-09' | album_release_date == '2006-10-24' |
           track_name == 'Sweeter Than Fiction - From "One Chance" Soundtrack') %>%
  filter(album_name != 'Spotify Singles' & !grepl('witch', album_name) & !grepl('Remix', track_name) & !grepl('the long pond studio sessions', track_name) & !grepl('Live From Paris', track_name) & !grepl('Video Edition', track_name) & !grepl('Live from the 2020 Academy of Country Music Awards', track_name) & !grepl('Recorded Live at the 2019 iHeartRadio Jingle Ball', track_name) & !grepl('cabin in candlelight version', track_name) & !grepl('Love Story - Pop Mix', track_name) & !grepl('Sawyr And Ryan Tedder Mix', track_name) & !grepl("willow - 90's trend remix", track_name) & !grepl('Acoustic Version', track_name) & !grepl('Piano', track_name)) %>%
  distinct(track_name, .keep_all = T)

ts[ts$album_name == 'folklore: the long pond studio sessions (from the Disney+ special) [deluxe edition]', 'album_name'] <- 'folklore (deluxe version)'
knitr::kable(sample_n(ts, 13), col.names = gsub("[.]", " ", names(ts))) %>% 
  kable_styling(full_width = F) %>% 
  scroll_box(width = "100%", height = "350px")
searchTrack <- function(artist, songName, api_key) {
  require(jsonlite)
  artist <- gsub(" ", "%20", artist)
  song <- gsub(" ", "%20", songName)
  data <- jsonlite::fromJSON(paste0(
    "http://api.musixmatch.com/ws/1.1/track.search?q_artist=", artist, "&q_track=", song, "&apikey=", api_key))
  
  trackInfo <- data.frame(track_id = unlist(data$message$body$track_list$track$track_id),
                       track_name = data$message$body$track_list$track$track_name,
                       album_name = data$message$body$track_list$track$album_name,
                       album_id = data$message$body$track_list$track$album_id,
                       artist_name = data$message$body$track_list$track$artist_name,
                       explicit = data$message$body$track_list$track$explicit,
                       stringsAsFactors = FALSE)
  return(trackInfo)
} #end
getLyrics <- function(trackID, api_key){
  require(jsonlite)
  metadata <- jsonlite::fromJSON(paste0(
    "http://api.musixmatch.com/ws/1.1/track.lyrics.get?track_id=", trackID, "&apikey=", api_key))

tracks <- data.frame(track_id = trackID,
                     lyrics_body = gsub("[\r\n]", " ",  metadata$message$body$lyrics$lyrics_body))
  return(tracks)
} # end
# We create a `test` data frame which holds all of the track names and track id's in Musixmatch's catalog
test <- data.frame()
for (i in ts$track_name) {
  test <- rbind(test, searchTrack('taylor swift', i, api_key))
}

# We check what tracks are stored with different naming conventions between the two datasets
test %>%
  filter(!(track_name %in% ts$track_name)) %>%
  distinct(track_name)

# we filter out the repeated track titles (and also songs in Musixmatch that are covers or instrumental versions which we don't want to consider)
test <- test %>%
  filter((track_name %in% ts$track_name)) %>%
  distinct(track_name, .keep_all = T) 

test <- test %>% 
  rbind(test, searchTrack('taylor swift', 'Sweeter Than Fiction', api_key)) %>% 
  rbind(test, searchTrack('taylor swift', "it's time to go (bonus track)", api_key)) %>% 
  rbind(test, searchTrack('taylor swift', "right where you left me (bonus track)", api_key)) %>% 
  rbind(test, searchTrack('taylor swift', "Only The Young (Featured in Miss Americana)", api_key)) %>% 
  rbind(test, searchTrack('taylor swift', "'tis the damn season", api_key)) %>% 
  rbind(test, searchTrack('taylor swift', "Lavender Haze (Felix Jaehn Remix)", api_key)) %>% 
  rbind(test, searchTrack('taylor swift', "Nothing New (Taylor's Version) (From The Vault) [feat. Phoebe Bridgers]", api_key)) %>% 
  rbind(test, searchTrack('taylor swift', "I Bet You Think About Me (Taylor's Version) (From The Vault) [feat. Chris Stapleton]", api_key)) %>% 
  distinct(track_name, .keep_all = T) %>% 
  filter(track_id != 269664185 & track_id != 253984170) %>% 
  filter(track_id != 258853439 & track_id != 226322338 & track_id != 258853440)

# we use our getLyrics() function to find the lyrics using the track_id in our `test` dataset 
lyrics <- data.frame()
for (i in test$track_id) {
  lyrics <- rbind(lyrics, getLyrics(i, api_key))
}
knitr::kable(head(test, 6)) %>% 
  kable_styling(full_width = F) %>% 
  scroll_box(width = "100%", height = "350px")
knitr::kable(head(lyrics, 6)) %>% 
  kable_styling(full_width = F) %>% 
  scroll_box(width = "100%", height = "350px")
# we merge test and lyrics
lyrics <- merge(lyrics, test, by = 'track_id')

# we check the naming conventions between each dataset
lyrics %>%
  filter(!(track_name %in% ts$track_name))
ts %>%
  filter(!track_name %in% lyrics$track_name)

# we fix the naming conventions (we adhere to the names that they are given in the Spotify catalog)
lyrics[lyrics$track_name == "Electric Touch (Taylor’s Version) (From The Vault) [feat. Fall Out Boy]", 'track_name'] <- "Electric Touch (feat. Fall Out Boy) (Taylor’s Version) (From The Vault)"
lyrics[lyrics$track_name == "Castles Crumbling (Taylor’s Version) (From The Vault) [feat. Hayley Williams]", 'track_name'] <- "Castles Crumbling (feat. Hayley Williams) (Taylor’s Version) (From The Vault)"
lyrics[lyrics$track_name == "Run (Taylor's Version) (From The Vault) [feat. Ed Sheeran]", 'track_name'] <- "Run (feat. Ed Sheeran) (Taylor’s Version) (From The Vault)"
lyrics[lyrics$track_name == "I Bet You Think About Me (Taylor's Version) [From The Vault] [feat. Chris Stapleton]", 'track_name'] <- "I Bet You Think About Me (feat. Chris Stapleton) (Taylor’s Version) (From The Vault)"
lyrics[lyrics$track_name == "Nothing New (Taylor's Version) (From The Vault) [feat. Phoebe Bridgers]", 'track_name'] <- "Nothing New (feat. Phoebe Bridgers) (Taylor’s Version) (From The Vault)"
lyrics[lyrics$track_name == "Everything Has Changed (Taylor's Version) [feat. Ed Sheeran]", 'track_name'] <- "Everything Has Changed (feat. Ed Sheeran) (Taylor’s Version)"
lyrics[lyrics$track_name == "The Last Time (Taylor's Version) [feat. Gary Lightbody]", 'track_name'] <- "The Last Time (feat. Gary Lightbody of Snow Patrol) (Taylor’s Version)"
lyrics[lyrics$track_name == "it's time to go (bonus track)", 'track_name'] <- "it’s time to go - bonus track"
lyrics[lyrics$track_name == "right where you left me (bonus track)", 'track_name'] <- "right where you left me - bonus track"
lyrics[lyrics$track_name == "'tis the damn season", 'track_name'] <- "‘tis the damn season"
lyrics[lyrics$track_name == "Only The Young (Featured in Miss Americana)", 'track_name'] <- "Only The Young - Featured in Miss Americana"
lyrics[lyrics$track_name == "Sweeter Than Fiction", 'track_name'] <- 'Sweeter Than Fiction - From "One Chance" Soundtrack'  
lyrics[lyrics$track_name == "Lavender Haze (Felix Jaehn Remix)", 'track_name'] <- 'Lavender Haze'
lyrics[lyrics$track_name == "   
Message In A Bottle (Taylor's Version) [From The Vault]", 'track_name'] <- "Message In A Bottle (Taylor's Version) (From The Vault)"
lyrics[lyrics$track_name == "Forever Winter (Taylor's Version) [From The Vault]", 'track_name'] <- "Forever Winter (Taylor's Version) (From The Vault)"

# we finally curate our official taylor swift catalog, with the track name, album name, track number, lyrics, and length
official_list <- merge(lyrics, ts, by = 'track_name') %>%
  select('track_name', 'album_name.y', 'track_number', 'lyrics_body', 'duration_ms')
official_list <- read.csv("~/Desktop/Project/official_list.csv")
official_list <- official_list[,-1]
knitr::kable(sample_n(official_list, 13)) %>% 
  kable_styling(full_width = F) %>% 
  scroll_box(width = "100%", height = "400px")
vis_miss(official_list)
ggplot(official_list %>% 
  mutate(era = ifelse("Taylor Swift" == official_list$album_name.y, "Self-titled", ifelse(grepl("Fearless", official_list$album_name.y), "Fearless", 
  ifelse(grepl("Speak Now", official_list$album_name.y), "Speak Now",  
  ifelse(grepl("Red", official_list$album_name.y) | grepl("Message In A Bottle", official_list$album_name.y) | grepl("All Too Well", official_list$album_name.y), "RED", 
  ifelse(grepl("1989", official_list$album_name.y) | grepl("Sweeter Than Fiction", official_list$album_name.y), "1989",
  ifelse(grepl("reputation", official_list$album_name.y), "reputation", 
  ifelse(grepl("Christmas Tree Farm", official_list$album_name.y) | grepl("Lover", official_list$album_name.y) | grepl("Americana", official_list$album_name.y) | grepl("The Cruelest Summer", official_list$album_name.y) | grepl("All Of The Girls You Loved Before", official_list$album_name.y), "Lover", 
  ifelse(grepl("folklore", official_list$album_name.y) | grepl("Carolina", official_list$album_name.y) | grepl("the lakes", official_list$album_name.y) | grepl("Renegade", official_list$album_name.y), "folklore", 
  ifelse(grepl("evermore", official_list$album_name.y) | grepl("The Joker And The Queen", official_list$album_name.y), "evermore",  
         ifelse(grepl("Midnights", official_list$album_name.y) | grepl("You're Losing Me", official_list$album_name.y) | grepl("Anti-Hero", official_list$album_name.y), "Midnights", NA))))))))))) %>% 
  mutate(era = factor(era, levels = c("Self-titled", "Fearless", "Speak Now", "RED", "1989", "reputation", "Lover", "folklore", "evermore", "Midnights"))) %>% 
  arrange(era), aes(x= era, fill = era)) +
  geom_bar(stat = "count") + 
  theme_linedraw() + 
  theme(axis.ticks.x = element_blank(), axis.ticks.y = element_blank(), axis.text.x = element_text(angle = 90, vjust = 0.5, hjust = 0.5), legend.position = "none") + 
  scale_y_continuous(expand=c(0,0)) +
  scale_x_discrete(labels = c("Self-titled", "Fearless", "Speak Now", "RED", "1989", "reputation", "Lover", "folklore", "evermore", "Midnights")) +
  scale_fill_manual(values = c("#b9d2b5", "#f4cb8d", "#d1b2d2", "#823549", "#b5e9f6", "#847e80", "#f9b2d0", "#cfcac6", "#c8ae95", "#434961")) +
  labs(title = "Taylor Swift Songs per Era", x = "Era",
       y = element_blank()) +
  coord_flip()
ts_stopwords <- as_tibble(stopwords::stopwords(source = 'snowball')) %>%
  filter(!(value %in% c("me", "our", "ours", "your", "did", "would", "could", "should", "before", "after", "between", "against", "above", "below", "up", "down", "again", "why", "only", "same", "too", "will", "myself")))

lyricFreq <- function(index) {
  # this is to find when the lyrics end so we don't have to deal with the copyright message
  end <- which(strsplit(official_list$lyrics_body[index], split = ' ')[[1]] == "...")
  # to split up the lyric text by word 
  rv <- strsplit(tolower(official_list$lyrics_body[index]), split = ' ')[[1]][1:(end-1)] 
  rv <- gsub('[,()"!?]', '', rv)

  # this is a data frame of all the frequent words (in order)
  #return(count(rv)[order(count(rv)$freq, decreasing = T),])
  return(as_tibble(rv) %>%
  filter(!(value %in% as.vector(ts_stopwords)$value)) %>%
  mutate(stem = wordStem(value)) %>%
  count(stem) %>%
  arrange(desc(n)))
}
official_list$lyrics_body[153]

knitr::kable(lyricFreq(153)) %>% 
  kable_styling(full_width = T) %>% 
  scroll_box(width = "30%", height = "200px")

wordcloud(words = lyricFreq(153)$stem, 
          freq = lyricFreq(153)$n, 
          min.freq = 1,
          max.words = 200,
          colors = brewer.pal(8, "Dark2"))
official_list$lyrics_body[218]

knitr::kable(lyricFreq(218)) %>% 
  kable_styling(full_width = T) %>% 
  scroll_box(width = "30%", height = "200px")

wordcloud(words = lyricFreq(218)$stem, 
          freq = lyricFreq(218)$n, 
          min.freq = 2,
          max.words = 200,
          colors = brewer.pal(3, "Dark2"))
official_list$lyrics_body[9]

knitr::kable(lyricFreq(9)) %>% 
  kable_styling(full_width = T) %>% 
  scroll_box(width = "30%", height = "200px")

wordcloud(words = lyricFreq(9)$stem, 
          freq = lyricFreq(9)$n, 
          min.freq = 1,
          max.words = 200,
          colors = brewer.pal(8, "Set1"))
myDict <- dictionary(list(love = c('love*', 'girl*', 'forev*', 'light*', 'first', 'summer', 'time', 'friend', 'eye', 'see', 'bab*', 'romeo', 'run', 'one', 'spark*', 'wonder*', 'enchant*', 'handsom*', 'best', 'daydream', 'treacherous', 'fearless', 'style', 'grin*', 'electr*', 'flawless', 'hand', 'craz*', 'favorit*', 'lip*', 'blind', 'book', 'pac*', 'town', 'sunshin*', 'dress', 'breez', 'dream', 'fairytal*', 'save', 'tall', 'somedai', 'smile', 'laugh', 'togeth', 'spark', 'belong', 'girlfriend', 'red', 'invisib*', 'begin', 'wild*', 'magic*', 'timeless', 'boy*', 'wildest', 'heaven', 'sunset', 'arms', 'different', 'silence', 'sweet', 'skin', 'high', 'angel', 'glow', 'shape', 'bad', 'ring'),
                          pain = c('remember', 'never', 'lost', 'lose', 'losin*', 'forget', 'dead', 'cruel', 'fall', 'right', 'wait', 'sorry', 'swallow', 'blurry', 'worth', 'breathe', 'gray', 'shade', 'away', 'sad', 'tragic', 'tear*', 'richochet', 'last', 'stream*', 'mess*', 'cri*', 'paralyz*', 'kill*', 'exile', 'trouble', 'problem*', 'final', 'champagn', 'crestfallen', 'worst', 'infidel', 'broken', 'regret', 'wound', 'ghost', 'poison', 'scare', 'weapon', 'pain', 'reveng', 'fake', "ain't", "should'v", 'perfectli', 'apart', 'sign', 'complic', 'break', 'broke', 'empti', 'speak', 'haunt*', 'last', 'breakd*', 'think', 'mcgraw', 'fade', 'shame', 'toler*', 'better', 'desper*', "could'v", "would'v", 'apologi*', 'door', 'fight', 'god', 'screaming', 'woods', 'sad', 'without', 'tire*', 'called', 'car', 'wash', 'stupid', 'anymor', 'storm'),
                          change = c('face', 'year*', 'change', 'happi', 'free', 'confus*', 'same', 'revolution', 'chance', 'day*', 'grow', 'grew', 'young', 'littl*', 'guess*', 'live', 'pretend', 'york', 'welcom', 'waitin*', 'danc*', 'forevermor', 'loud', 'ash', 'best', 'fireplac*', 'learn', 'sprinkler', 'know', 'year', 'new', 'proud*', '22', 'alright', 'crowd', 'creek', 'pennsylvania', 'innoc*', 'balanc*', 'bigger', 'combat', 'easi*', 'archer', 'win', 'darl*', 'shake', 'peopl*', 'world', 'city'),
                          reflection = c('hero', 'reputation', 'dynasti', 'power', 'play*', 'man', 'rude', 'mirrorball', 'version', 'believe', 'natural', 'try', 'empir', 'bridg*', 'castle*', 'watch', 'everybod*', 'fallout', 'kill', 'proof', 'woman', 'women', 'gold', 'american*', 'everybod*', 'depress*', 'anti-hero', 'game', 'underlin*', 'cynic*', 'hunter', 'peak', 'nothin*', 'karma', 'street', 'vice', 'wise*')))
ts_dtm2 <- tokens(corpus(official_list$lyrics_body)) %>% 
  tokens_remove(pattern = ts_stopwords) %>% 
  tokens_tolower() %>% 
  tokens_remove(pattern = c('lyrics', 'commercial', 'use', '1409624132591'))

ts_dtm2 <- ts_dtm2 %>% dfm()
knitr::kable(dfm_sample(ts_dtm2, 6)) %>% 
  kable_styling(full_width = F) %>% 
  scroll_box(width = "100%", height = "250px")
ts_dfm <- dfm_lookup(ts_dtm2, dictionary = myDict)

ts_dfm <- ts_dfm %>% as.data.frame() %>% 
  cbind(official_list$track_name) %>% 
  select(-c('doc_id')) %>% 
  select(c(5, 1:4))
knitr::kable(ts_dfm) %>% 
  kable_styling(full_width = F) %>% 
  scroll_box(width = "100%", height = "400px")
set.seed(13)
kmeans(ts_dfm[,-1], centers = 2)
set.seed(13)
kmeans(ts_dfm[,-1], centers = 3)
set.seed(13)
kmeans(ts_dfm[,-1], centers = 4)
set.seed(13)
kmeans(ts_dfm[,-1], centers = 5)
set.seed(13)
kmeans_ts1 <- kmeans(ts_dfm[,-1], centers = 4)
kmeans_ts2 <- kmeans(ts_dfm[,-1], centers = 4)
kmeans_ts3 <- kmeans(ts_dfm[,-1], centers = 4)
kmeans_ts4 <- kmeans(ts_dfm[,-1], centers = 4)
kmeans_ts5 <- kmeans(ts_dfm[,-1], centers = 4)
kmeans_ts6 <- kmeans(ts_dfm[,-1], centers = 4)
kmeans_ts1$centers
kmeans_ts2$centers
kmeans_ts3$centers
kmeans_ts4$centers
kmeans_ts5$centers
kmeans_ts6$centers
# plot for versions: kmeans_2, kmeans_3, kmeans_5
ggplot(cbind(ts_dfm, kmeans_ts2$cluster), aes(x = love, y = pain, color = factor(kmeans_ts2$cluster))) +
  geom_point(size = 0.9, shape = 6) + 
  scale_color_manual(limits = c(2, 4, 3, 1), values = c("pink", "navy", "purple", "gray"), name = "Theme", labels = c("Love", "Pain", "Change", "NA")) +
  theme_light() + 
  theme(axis.ticks.x = element_blank(), axis.ticks.y = element_blank(), plot.title = element_text(size=11.5, face = "bold"),     legend.position = c(1.01, 1.02),
    legend.justification = c("right", "top"),
    legend.box.just = "right",
    legend.margin = margin(6, 6, 6, 6),
    legend.text = element_text(size = 6),
    legend.box.background = element_rect(color="black", size=0.25)) +
  labs(title = "Frequency of Love vs Pain Words by Theme", 
       x = "Love", y = "Pain")

# plot for versions: kmeans_1
ggplot(cbind(ts_dfm, kmeans_ts1$cluster), aes(x = love, y = pain, color = factor(kmeans_ts1$cluster))) +
  geom_point(size = 0.9, shape = 6) + 
  scale_color_manual(limits = c(2, 3, 1, 4), values = c("pink", "navy", "purple", "gray"), name = "Theme", labels = c("Love", "Pain", "Change", "NA")) +
  theme_light() + 
  theme(axis.ticks.x = element_blank(), axis.ticks.y = element_blank(), plot.title = element_text(size=11.5, face = "bold"),     legend.position = c(1.01, 1.02),
    legend.justification = c("right", "top"),
    legend.box.just = "right",
    legend.margin = margin(6, 6, 6, 6),
    legend.text = element_text(size = 6),
    legend.box.background = element_rect(color="black", size=0.25)) +
  labs(title = "Frequency of Love vs Pain Words by Theme", 
       x = "Love", y = "Pain")

# plot for versions: kmeans_4
ggplot(cbind(ts_dfm, kmeans_ts4$cluster), aes(x = love, y = pain, color = factor(kmeans_ts4$cluster))) +
  geom_point(size = 0.9, shape = 6) + 
  scale_color_manual(limits = c(1, 3, 2, 4), values = c("pink", "navy", "gold", "gray"), name = "Theme", labels = c("Love", "Pain", "Reflection", "NA")) +
  theme_light() + 
  theme(axis.ticks.x = element_blank(), axis.ticks.y = element_blank(), plot.title = element_text(size=11.5, face = "bold"),     legend.position = c(1.01, 1.02),
    legend.justification = c("right", "top"),
    legend.box.just = "right",
    legend.margin = margin(6, 6, 6, 6),
    legend.text = element_text(size = 6),
    legend.box.background = element_rect(color="black", size=0.25)) +
  labs(title = "Frequency of Love vs Pain Words by Theme", 
       x = "Love", y = "Pain")

# plot for version: kmeans_6
ggplot(cbind(ts_dfm, kmeans_ts6$cluster), aes(x = love, y = pain, color = factor(kmeans_ts6$cluster))) +
  geom_point(size = 0.9, shape = 6) + 
  scale_color_manual(limits = c(1, 2, 4, 3), values = c("pink", "maroon", "navy", "gray"), name = "Theme", labels = c("Strong Love", "Moderate Love", "Pain", "NA")) +
  theme_light() + 
  theme(axis.ticks.x = element_blank(), axis.ticks.y = element_blank(), plot.title = element_text(size=11.5, face = "bold"),     legend.position = c(1.01, 1.02),
    legend.justification = c("right", "top"),
    legend.box.just = "right",
    legend.margin = margin(6, 6, 6, 6),
    legend.text = element_text(size = 6), 
    legend.box.background = element_rect(color="black", size=0.25)) +
  labs(title = "Frequency of Love vs Pain Words by Theme", 
       x = "Love", y = "Pain")
# create different hierarchical clusterings
hcluster_complete_ts <- hclust(dist(ts_dfm), method = "complete")
hcluster_single_ts <- hclust(dist(ts_dfm), method = "single")
hcluster_average_ts <- hclust(dist(ts_dfm), method = "average")
plot(hcluster_complete_ts, main = "Complete Linkage",
     xlab = "", sub = "", cex = 0.17)
plot(hcluster_single_ts, main = "Single Linkage",
     xlab = "", sub = "", cex = .17)
plot(hcluster_average_ts, main = "Average Linkage",
     xlab = "", sub = "", cex = .17)
cutree(hcluster_complete_ts, 4)
cutree(hcluster_single_ts, 4)
cutree(hcluster_average_ts, 4)
table(cutree(hcluster_complete_ts, 4))
knitr::kable(official_list[which(cutree(hcluster_complete_ts, 4) == 1), 1]) %>% 
  kable_styling(full_width = F) %>% 
  scroll_box(width = "30%", height = "200px")
knitr::kable(official_list[which(cutree(hcluster_complete_ts, 4) == 2), 1]) %>% 
  kable_styling(full_width = F) %>% 
  scroll_box(width = "30%", height = "200px")
knitr::kable(official_list[which(cutree(hcluster_complete_ts, 4) == 3), 1]) %>% 
  kable_styling(full_width = F) %>% 
  scroll_box(width = "30%", height = "200px")
knitr::kable(official_list[which(cutree(hcluster_complete_ts, 4) == 4), 1]) %>% 
  kable_styling(full_width = F) %>% 
  scroll_box(width = "30%", height = "200px")
cutree(hcluster_complete_ts, 5)
table(cutree(hcluster_complete_ts, 5))
knitr::kable(official_list[which(cutree(hcluster_complete_ts, 5) == 1), 1]) %>% 
  kable_styling(full_width = F) %>% 
  scroll_box(width = "30%", height = "200px")
knitr::kable(official_list[which(cutree(hcluster_complete_ts, 5) == 2), 1]) %>% 
  kable_styling(full_width = F) %>% 
  scroll_box(width = "30%", height = "200px")
knitr::kable(official_list[which(cutree(hcluster_complete_ts, 5) == 3), 1]) %>% 
  kable_styling(full_width = F) %>% 
  scroll_box(width = "30%", height = "200px")
knitr::kable(official_list[which(cutree(hcluster_complete_ts, 5) == 4), 1]) %>% 
  kable_styling(full_width = F) %>% 
  scroll_box(width = "30%", height = "200px")
knitr::kable(official_list[which(cutree(hcluster_complete_ts, 5) == 5), 1]) %>% 
  kable_styling(full_width = F) %>% 
  scroll_box(width = "30%", height = "200px")
cutree(hcluster_complete_ts, 6)
table(cutree(hcluster_complete_ts, 6))
knitr::kable(official_list[which(cutree(hcluster_complete_ts, 6) == 6), 1]) %>% 
  kable_styling(full_width = F) %>% 
  scroll_box(width = "30%", height = "200px")
hclust5_ts <- cutree(hcluster_complete_ts, 5)
ggplotly(ggplot(cbind(ts_dfm, kmeans_ts2$cluster), 
       aes(x = factor(kmeans_ts2$cluster), 
           fill = factor(kmeans_ts2$cluster), 
           text = official_list$track_name)) +
  geom_bar(stat = "count") +
  scale_x_discrete(limits = c(2, 4, 3, 1), labels = c("Pain", "NA", "Change", "Love")) +
  scale_fill_manual(limits = c(2, 4, 3, 1), values = c("pink", "navy", "purple", "gray"), name = "Theme", labels = c("Love", "Pain", "Change", "NA")) +
  theme_minimal() +
  theme(axis.title.y = element_blank())+
  labs(title = "K-Means Clustering: Songs per Theme", 
       x = "Theme"),
  tooltip = c("text")) %>% 
  layout(showlegend = F)

ggplotly(ggplot(cbind(ts_dfm, hclust5_ts), 
       aes(x = factor(hclust5_ts), 
           fill = factor(hclust5_ts), 
           text = official_list$track_name)) +
  geom_bar(stat = "count") +
  scale_x_discrete(limits = c(1, 4, 3, 5, 2), labels = c("Love", "Reflection", "Change", "NA", "Pain")) +
  scale_fill_manual(limits = c(1, 4, 3, 5, 2), values = c("pink", "navy", "purple", "gold", "gray"), name = "Theme", labels = c("Love", "Pain", "Change", "Reflection", "NA")) +
  theme_minimal() +
  theme(axis.title.y = element_blank())+
  labs(title = "Hierarchical Clustering: Songs per Theme", 
       x = "Theme"),
  tooltip = c("text")) %>% 
  layout(showlegend = F)
knitr::include_graphics(path = 'ttpd.png')

(W24) PSTAT 131: Taylor Swift Spotify/Musixmatch Project

An Unsupervised Learning Model & Text Analysis on Lyrical Data

Anthony Cu

18 March 2024